Aegis-GPU5.1 Hardware fault triage
Reading Xid errors, checking fans and pulling a suspect node out of production.
Key points
An Xid is an error report from the NVIDIA driver, written to the kernel log with a number. PCIe (PCI Express) is the bus that connects the GPU to the CPU. Xid 79 means the GPU vanished from that bus. Hardware faults on the PCIe link often cause it.
What NVIDIA says (2)
“This event is logged when the GPU driver attempts to access the GPU over its PCI Express connection and finds that the GPU is not accessible.”
“This event is often caused by hardware failures on the PCI Express link”
ECC (error-correcting code) memory can fix single-bit errors. A double bit error cannot be fixed. Xid 48 reports that uncorrectable error. NVIDIA says a GPU reset or node reboot is needed. nvidia-smi can summarize ECC errors.
What NVIDIA says (3)
“This event is logged when the GPU detects that an uncorrectable error occurs on the GPU.”
“A GPU reset or node reboot is needed to clear this error.”
“The tool nvidia-smi can provide a summary of ECC errors.”
A fan module holds two fans. If either fails, you replace the whole module. NVIDIA lists three ways to find it: the LED, nvsm show fans, and BMC sensor data.
What NVIDIA says (3)
“Run the nvsm show fans command and view the command output.”
“If a fan is running at an abnormal speed, then that fan needs to be replaced.”
“If either fan fails, then the entire module must be replaced.”
Triage means sorting a problem to decide what to do. A batch partition is the Slurm queue that runs normal jobs. Pulling the node out stops it from hurting more jobs while you test it.
What NVIDIA says (2)
“If an issue is found, it should be removed from the batch partition for initial triage.”
“For health checks, this is the NVIDIA System Management tool (nvsm).”
Key terms
- NVIDIA System Management: The DGX tool that reports system health and runs stress tests from one command line.
- Xid error: A numbered GPU driver error in the kernel log that points to a class of fault.
Try it
Sample question
The kernel log shows 'NVRM: Xid (PCI:0000:5a:00): 79, GPU has fallen off the bus.' What does this mean?
Show the answer
Answer: The driver tried to reach the GPU over PCI Express and found it inaccessible.
An Xid is an error report from the NVIDIA driver, written to the kernel log with a number. PCIe (PCI Express) is the bus that connects the GPU to the CPU. Xid 79 means the GPU vanished from that bus. Hardware faults on the PCIe link often cause it.
What NVIDIA says (2)
“This event is logged when the GPU driver attempts to access the GPU over its PCI Express connection and finds that the GPU is not accessible.”
“This event is often caused by hardware failures on the PCI Express link”
Practice 5.1 (4 questions) Full Troubleshoot and Optimize guide