1.4 Firmware upgrades and fault detection

Key points

1. Firmware is the low-level software stored on the board, such as the BMC and SBIOS. Secure Flash checks the image signature before writing it, so only signed, verified firmware is installed.

   What NVIDIA says (1)

   “Secure Flash is implemented for the DGX H100/H200 to prevent unsigned and unverified firmware images from being flashed onto the system.”

   — DGX H100/H200 User Guide: Security

2. OPN (ordering part number) identifies the card model. PSID (Parameter-Set Identification) is a 16-character ID in the firmware that marks the card configuration. The image must match both. After flashing, an AC power cycle makes the new firmware take effect. An AC power cycle means fully removing and restoring power.

   What NVIDIA says (3)

   “If MLNX_OFED is installed, use the mlxfwmanager tool to update the firmware.”

   — DGX H100/H200 Service Manual: Updating the ConnectX-7 Firmware

   “PSID (Parameter-Set Identification) is a 16-ascii character string embedded in the firmware image”

   — DGX H100/H200 Service Manual: Updating the ConnectX-7 Firmware

   “Perform an AC power cycle on the system for the firmware update to take effect.”

   — DGX H100/H200 Service Manual: Updating the ConnectX-7 Firmware

3. Each ConnectX port appears as an mlx5 device under /sys/class/infiniband. Its fw_ver file shows the running firmware. NVIDIA's step is to confirm all versions are the same.

   What NVIDIA says (2)

   “After the system starts, log in and confirm the firmware versions are all the same:”

   — DGX H100/H200 Service Manual: Updating the ConnectX-7 Firmware

   “cat /sys/class/infiniband/mlx5_*/fw_ver”

   — DGX H100/H200 Service Manual: Updating the ConnectX-7 Firmware

4. mlxfwmanager is NVIDIA's firmware update and query tool for network adapters. --query lists each device with its current and available firmware, and its status.

   What NVIDIA says (2)

   “The mlxfwmanager is a firmware update and query utility which scans the system for available NVIDIA devices (only mst PCI devices) and performs the necessary firmware updates.”

   — NVIDIA Firmware Tools: mlxfwmanager

   “To query all the devices on the machine, use the following command line: # mlxfwmanager --query”

   — NVIDIA Firmware Tools: mlxfwmanager

Key terms

• Secure Flash: A DGX protection that refuses firmware images that are not signed and verified.
• Parameter-Set Identification: A 16-character string in NIC firmware that must match the board, so the right image is flashed.
• mlxfwmanager: The NVIDIA tool that queries and updates firmware on NVIDIA network adapters.

Sample question

What protects a DGX H100 from someone flashing an unsigned firmware image?