Aegis-GPU4.2 Troubleshooting Fabric Manager
Service status, version match, logs and the not-ready error.
Key points
Fabric Manager (FM) configures the NVSwitch fabric so GPUs can talk over NVLink. It runs as a systemd service.
What NVIDIA says (1)
“To check FM, for Linux based OS distributions, run the following command: sudo systemctl status”
Without FM, the NVSwitch fabric is not set up, so new CUDA jobs are refused.
What NVIDIA says (1)
“A new CUDA job launch will fail with a cudaErrorSystemNotReady error.”
A mismatch between FM and the driver is a common cause of FM start failures.
What NVIDIA says (1)
“NVIDIA Fabric Manager Package (same version as the Driver package).”
Check this log first when FM fails to start or reports NVSwitch errors.
What NVIDIA says (1)
“Default Value LOG_FILE_NAME=/var/log/fabricmanager.log”
Key terms
- Fabric Manager: The service that configures NVSwitches so GPUs can talk over NVLink; its version must match the driver.
Sample question
On an NVSwitch system, how do you check whether Fabric Manager is running?
Show the answer
Answer: sudo systemctl status nvidia-fabricmanager
Fabric Manager (FM) configures the NVSwitch fabric so GPUs can talk over NVLink. It runs as a systemd service.
What NVIDIA says (1)
“To check FM, for Linux based OS distributions, run the following command: sudo systemctl status”
Practice 4.2 (4 questions) Full Troubleshooting and Optimization guide
← 4.1 Troubleshooting Docker with GPUs · 4.3 Troubleshooting BCM →