4.1 Single-node stress test

Key points

1. DCGM (Data Center GPU Manager) diagnostics come in run levels. A higher level runs longer and includes every test below it. Level 1 is the quick readiness check. Level 4 is the extended, longer-running hardware diagnostic.

   What NVIDIA says (3)

   “higher numbered tests include all beneath.”

   — DCGM User Guide: DCGM Diagnostics

   “4 - Extended (Longer-running System HW Diagnostics)”

   — DCGM User Guide: DCGM Diagnostics

   “Level 1 tests to use as a readiness metric”

   — DCGM User Guide: DCGM Diagnostics

2. A single-node stress test checks one server alone, before testing the fabric. On DGX, NVIDIA runs it with NVSM.

   What NVIDIA says (2)

   “To run the tests, use NVSM.”

   — DGX H100/H200 Service Manual: Introduction

   “NVIDIA recommends running the pre-flight stress test before putting a system into a production environment or after servicing.”

   — DGX H100/H200 Service Manual: Introduction

Key terms

• NVIDIA System Management: The DGX tool that reports system health and runs stress tests from one command line.
• DCGM diagnostics: dcgmi diag tests GPU health at run levels 1 to 4; higher levels run longer and include the lower ones.

Sample question

You want DCGM's longest built-in hardware diagnostic on one node before you hand it to users. Which run level do you pick?