Aegis-GPU

4.6 Switch firmware and software

NCP-AII · Cluster Test and Verification (33% of the exam) · Official objective: “Confirm FW/SW on switches.”

Checking switch versions are aligned with UFM or flint.

Key points

  1. Version drift means devices run different releases. That can cause odd behavior. The guide checks switch ASICs, transceivers and HCAs, and recommends UFM for the check. An HCA (host channel adapter) is the InfiniBand NIC in a server.

    What NVIDIA says (2)

    “The recommended guideline is to confirm that the versions among the cluster are aligned, or differ with up to 2 versions.”

    — InfiniBand Cluster Bring-up Procedure: Confirm Components' Firmware and Software Versions

    “The process can be done using UFM GUI ( which is recommended), or through MOFED commands.”

    — InfiniBand Cluster Bring-up Procedure: Confirm Components' Firmware and Software Versions

  2. An unmanaged switch has no switch OS of its own. You reach it in-band through its LID. A LID (local identifier) is the switch's InfiniBand address. On managed switches, the ASIC firmware is bundled in the switch OS.

    What NVIDIA says (2)

    “This section is applicable only to externally managed (unmanaged) switches (the ASIC firmware is bundled in NOS in managed systems).”

    — InfiniBand Cluster Bring-up Procedure: Confirm Components' Firmware and Software Versions

    “Check the firmware version, run flint -d lid-X -qq q”

    — InfiniBand Cluster Bring-up Procedure: Confirm Components' Firmware and Software Versions

  3. UFM is NVIDIA's fabric manager for InfiniBand. It monitors the fabric and pushes switch software. The SuperPOD guide upgrades InfiniBand switches through the UFM appliance.

    What NVIDIA says (2)

    “The high-speed InfiniBand fabrics are managed with NVIDIA Unified Fabric Manager (UFM).”

    — DGX SuperPOD Administration Guide: Managing High-Speed Fabrics

    “Check out the instructional guide on upgrading InfiniBand switches using the UFM appliance.”

    — DGX SuperPOD Deployment Guide: Upgrade InfiniBand Switches

  4. The UFM fabric health report runs a series of checks and lists errors and warnings. The bring-up guide expects every section green before moving on.

    What NVIDIA says (2)

    “UFM fabric health report contains the results of a series of checks that run on the fabric.”

    — InfiniBand Cluster Bring-up Procedure: UFM Fabric Health

    “Confirm that all fields are indicating green status”

    — InfiniBand Cluster Bring-up Procedure: UFM Fabric Health

Key terms

Sample question

NVIDIA's bring-up guide sets a rule for firmware and software versions across the cluster. What is it?

Show the answer

Answer: Versions should be aligned, or differ by up to 2 versions.

Version drift means devices run different releases. That can cause odd behavior. The guide checks switch ASICs, transceivers and HCAs, and recommends UFM for the check. An HCA (host channel adapter) is the InfiniBand NIC in a server.

What NVIDIA says (2)

“The recommended guideline is to confirm that the versions among the cluster are aligned, or differ with up to 2 versions.”

— InfiniBand Cluster Bring-up Procedure: Confirm Components' Firmware and Software Versions

“The process can be done using UFM GUI ( which is recommended), or through MOFED commands.”

— InfiniBand Cluster Bring-up Procedure: Confirm Components' Firmware and Software Versions

Practice 4.6 (4 questions) Full Cluster Test and Verification guide

← 4.5 Confirming cabling · 4.7 BlueField-3 firmware and software →