4.3 Troubleshooting BCM

/var/log/cmdaemon. CMDaemon is the BCM service on every node that carries out management commands.

NCP-AIO · Troubleshooting and Optimization (23% of the exam) · Official objective: “Troubleshoot Base Command Manager”

CMDaemon logs and debug mode, provisioning states and status.

Key points

CMDaemon is the BCM service on every node that carries out management commands.
What NVIDIA says (1)
“CMDaemon generates log messages in /var/log/cmdaemon from specific internal subsystems”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)
Debug output is off by default because it grows the log quickly. Turn it off when done.
What NVIDIA says (1)
“A global debug mode can be enabled in CMDaemon using cmdaemonctl”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)
The node-installer is the small system that provisions a node's image at boot.
What NVIDIA says (1)
“This state is entered from the INSTALLING state when the head node CMDaemon can no longer ping the node. It could indicate the node has crashed while running the node-installer.”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)
Provisioning copies software images to nodes. This command shows its queue and state.
What NVIDIA says (1)
“provisioningstatus Provisioning subsystem status Pending request: node001, node002”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)

Base Command Manager: NVIDIA cluster management software that provisions nodes from software images, monitors them and installs workload managers.
CMDaemon: The BCM daemon on each node that carries out management commands and writes /var/log/cmdaemon.

Where does the BCM cluster management daemon, CMDaemon, write its log messages?

Show the answer

Answer: /var/log/cmdaemon

CMDaemon is the BCM service on every node that carries out management commands.

What NVIDIA says (1)

“CMDaemon generates log messages in /var/log/cmdaemon from specific internal subsystems”
— NVIDIA Base Command Manager 11 Administrator Manual (PDF)