Aegis-GPU4.5 Confirming cabling
Comparing real cabling with the planned topology using cable validation and IB discovery tools.
Key points
Topology is the planned map of which port connects to which. The cable validation tool runs as a UFM plugin. It uses managed switches over the management network, so it works without a running subnet manager.
What NVIDIA says (3)
“Cable validation is the process of validating the actual cable deployment, against the expected topology (from the planning).”
“The tool is not dependent on a working SM, or any working IB communication at all, but utilizes the management interfaces of all the connected network devices.”
“The validation can utilize managed switches only.”
A GUID (globally unique identifier) names each InfiniBand device, like a MAC address. ibnetdiscover walks the subnet and prints every node and link.
What NVIDIA says (1)
“Performs InfiniBand subnet discovery and outputs a human readable topology file. GUIDs, node types, and port numbers are displayed as well as port LIDs and node descriptions.”
iblinkinfo shows each port's link, so you can spot links down or at the wrong width or speed.
What NVIDIA says (1)
“Reports link info for each port in an InfiniBand fabric, node by node.”
Each managed switch knows its neighbors on every port. The tool's agents collect this and compare it with the planned topology file. It repeats the check, so you can validate the cluster as you build it.
What NVIDIA says (2)
“Perform IB neighbors searchCompare the result to the expected topologyReport it to the main tool”
“This allows to bring up the cluster gradually / incrementally, and to validate the deployment in smaller pieces, rather than all at once.”
Key terms
- Cable validation: Comparing the real cabling, read from managed switches, with the planned topology.
- ibdiagnet: An InfiniBand tool that scans the whole fabric and reports connectivity, devices and link width and speed.
Try it
Sample question
What does NVIDIA's cable validation tool compare?
Show the answer
Answer: The actual cable deployment against the expected topology from planning.
Topology is the planned map of which port connects to which. The cable validation tool runs as a UFM plugin. It uses managed switches over the management network, so it works without a running subnet manager.
What NVIDIA says (3)
“Cable validation is the process of validating the actual cable deployment, against the expected topology (from the planning).”
“The tool is not dependent on a working SM, or any working IB communication at all, but utilizes the management interfaces of all the connected network devices.”
“The validation can utilize managed switches only.”
Practice 4.5 (4 questions) Full Cluster Test and Verification guide
← 4.4 Signal quality on cables · 4.6 Switch firmware and software →