1.7 Diagnosing cluster issues

Q: Support asks for cluster diagnostics from BCM. Which tool, run on the head node, gathers them?

cm-diagnose. cm-diagnose collects cluster data that helps diagnose issues. You can narrow its options for targeted collection.

NCP-AIO · Installation and Deployment (31% of the exam) · Official objective: “Diagnose and resolve cluster issues, such as job failures, node outages, or resource bottlenecks, using BCM.”

Health checks and the cm-diagnose tool for support cases.

Key points

cm-diagnose collects cluster data that helps diagnose issues. You can narrow its options for targeted collection.
What NVIDIA says (1)
“The diagnostic utility cm-diagnose is run from the head node. It gathers data on the cluster that may help diagnose issues.”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)
Health checks are BCM tests that report PASS or FAIL for a node or service. latesthealthdata lists the latest results.
What NVIDIA says (1)
“In cmsh, the statuses of the services are listed by running the latesthealthdata command (section 10.6.3) from device mode.”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)

Key terms

Health check: A BCM test that marks a node fit or unfit; a node failing a prejob check is kept from running the job.
cm-diagnose: The BCM utility, run on the head node, that gathers cluster data for support.

Sample question

Support asks for cluster diagnostics from BCM. Which tool, run on the head node, gathers them?

Show the answer

Answer: cm-diagnose

cm-diagnose collects cluster data that helps diagnose issues. You can narrow its options for targeted collection.

What NVIDIA says (1)

“The diagnostic utility cm-diagnose is run from the head node. It gathers data on the cluster that may help diagnose issues.”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)

Practice 1.7 (2 questions) Full Installation and Deployment guide

← 1.6 Node, DPU and switch networking · 1.8 Node categories →