1.10 Validating hardware for workloads

Key points

1. Validating hardware for workloads means proving a node can run real jobs. DCGM diagnostics have levels with different run times. Level 1 is the quick readiness metric. Level 2 suits a job epilogue after a failure. Levels 3 and 4 are for an administrator's post-mortem.

   What NVIDIA says (1)

   “Level 1 tests to use as a readiness metric Level 2 tests to use as an epilogue on failure Level 3 and Level 4 tests to be run by an administrator as post-mortem”

   — DCGM User Guide: DCGM Diagnostics

2. A stress test runs the hardware at high load to expose weak parts. NVSM's stress-test can target GPUs, CPU, memory and storage, for a set duration.

   What NVIDIA says (2)

   “NVIDIA recommends running the pre-flight stress test before putting a system into a production environment or after servicing.”

   — DGX H100/H200 Service Manual: Introduction

   “You can specify running the test on the GPUs, CPU, memory, and storage, and also specify the duration of the tests.”

   — DGX H100/H200 Service Manual: Introduction

3. A soak test keeps hardware under load for longer to expose intermittent faults. DCGM's --iterations runs the same suite in a loop.

   What NVIDIA says (2)

   “DCGM also supports running tests suites in loops using the --iterations option. Using this option allows for increasing the runtime duration of the tests.”

   — DCGM User Guide: DCGM Diagnostics

   “dcgmi diag -r pcie --iterations 3”

   — DCGM User Guide: DCGM Diagnostics

Key terms

• DCGM diagnostics: dcgmi diag tests GPU health at run levels 1 to 4; higher levels run longer and include the lower ones.

Sample question

Which DCGM diagnostic level does NVIDIA intend as a quick readiness check before a workload starts?