Aegis-GPUPhysical Layer Management
5% of the NCP-AII exam. Read each objective's key points, open “What NVIDIA says” to see the source, then practise.
System and Server Bring-up · Physical Layer Management · Control Plane Installation and Configuration · Cluster Test and Verification · Troubleshoot and Optimize
2.1 BlueField platform
NIC mode vs DPU mode, and installing a BFB image through RShim.
Key points
BlueField is NVIDIA's DPU (data processing unit): a network card with its own Arm CPU cores. In DPU Mode the Arm cores run and manage the NIC. In NIC Mode, BlueField behaves like a ConnectX adapter and, on BlueField-3, the Arm cores are inactive.
What NVIDIA says (2)
“In NIC Mode, BlueField operates as a ConnectX network adapter for the external host. For BlueField-3, the Arm cores are inactive”
“enforcing restrictions on the external host (host isolation)”
A BFB (BlueField bootstream) is the bundle with the BlueField OS and firmware. RShim is the host-side driver that gives a path into the BlueField over PCIe. Each BlueField shows up as rshim0, rshim1, and so on.
What NVIDIA says (2)
“host# sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb>”
“To list the RShim devices present on your system, run the following command:”
DPU mode turns on the BlueField's Arm cores. They run services that own the NIC and the data path. All host traffic flows through a virtual switch on those cores. In NIC mode the Arm cores are off.
What NVIDIA says (2)
“In DPU Mode, the NIC resources and functionality are owned and controlled by the embedded Arm subsystem.”
“BlueField-2 DPU and BlueField-3 DPU SKUs are shipped in DPU Mode by default.”
Key terms: BlueField DPU BFB image RShim
2.2 Configuring MIG
Enabling MIG mode, creating GPU and compute instances, and the DGX services that hold the GPU.
Key points
MIG (Multi-Instance GPU) splits one GPU into isolated instances. MIG mode is not enabled by default. Enabling it may need a GPU reset. If monitoring agents hold the GPU, the reset cannot happen and MIG stays pending. On DGX systems, NVIDIA's guide says to stop the nvsm and dcgm services, enable MIG on the GPU, and then restore those services.
What NVIDIA says (3)
“By default, MIG mode is not enabled on the GPU.”
“When MIG is enabled on the GPU, depending on the GPU product, the driver will attempt to reset the GPU so that MIG mode can take effect.”
“In this specific DGX example, you would have to stop the nvsm and dcgm services, enable MIG mode on the desired GPU and then restore the monitoring services”
MIG (Multi-Instance GPU) splits a GPU into isolated GPU instances. A compute instance runs work inside a GPU instance. '-cgi' creates GPU instances from profile IDs or names. '-C' also creates the default compute instance in each one.
What NVIDIA says (2)
“sudo nvidia-smi mig -cgi 9,3g.20gb -C”
“Once the GPU instances are created, you need to create the corresponding Compute Instances (CI).”
MIG (Multi-Instance GPU) splits one GPU into isolated instances. Turning it on can need the GPU free of other processes. On DGX, monitoring agents such as nvsm and dcgm hold the GPU. Stop them, enable MIG, then restart them.
What NVIDIA says (2)
“Please first kill all processes using the device and retry the command or reboot the system to make MIG mode effective.”
“you would have to stop the nvsm and dcgm services, enable MIG mode on the desired GPU and then restore th”
Older GPUs, such as A100, needed a GPU reset before MIG mode applied. From the Hopper generation on, the change takes effect without a reset.
What NVIDIA says (1)
“Starting with the Hopper generation of GPUs, enabling MIG mode no longer requires a GPU reset to take effect”
Key terms: Multi-Instance GPU
Try it: MIG Partitioning