2.2 Configuring MIG

Key points

1. MIG (Multi-Instance GPU) splits one GPU into isolated instances. MIG mode is not enabled by default. Enabling it may need a GPU reset. If monitoring agents hold the GPU, the reset cannot happen and MIG stays pending. On DGX systems, NVIDIA's guide says to stop the nvsm and dcgm services, enable MIG on the GPU, and then restore those services.

   What NVIDIA says (3)

   “By default, MIG mode is not enabled on the GPU.”

   — MIG User Guide: Getting Started with MIG

   “When MIG is enabled on the GPU, depending on the GPU product, the driver will attempt to reset the GPU so that MIG mode can take effect.”

   — MIG User Guide: Getting Started with MIG

   “In this specific DGX example, you would have to stop the nvsm and dcgm services, enable MIG mode on the desired GPU and then restore the monitoring services”

   — MIG User Guide: Getting Started with MIG

2. MIG (Multi-Instance GPU) splits a GPU into isolated GPU instances. A compute instance runs work inside a GPU instance. '-cgi' creates GPU instances from profile IDs or names. '-C' also creates the default compute instance in each one.

   What NVIDIA says (2)

   “sudo nvidia-smi mig -cgi 9,3g.20gb -C”

   — MIG User Guide: Getting Started with MIG

   “Once the GPU instances are created, you need to create the corresponding Compute Instances (CI).”

   — MIG User Guide: Getting Started with MIG

3. MIG (Multi-Instance GPU) splits one GPU into isolated instances. Turning it on can need the GPU free of other processes. On DGX, monitoring agents such as nvsm and dcgm hold the GPU. Stop them, enable MIG, then restart them.

   What NVIDIA says (2)

   “Please first kill all processes using the device and retry the command or reboot the system to make MIG mode effective.”

   — MIG User Guide: Getting Started with MIG

   “you would have to stop the nvsm and dcgm services, enable MIG mode on the desired GPU and then restore th”

   — MIG User Guide: Getting Started with MIG

4. Older GPUs, such as A100, needed a GPU reset before MIG mode applied. From the Hopper generation on, the change takes effect without a reset.

   What NVIDIA says (1)

   “Starting with the Hopper generation of GPUs, enabling MIG mode no longer requires a GPU reset to take effect”

   — MIG User Guide: Getting Started with MIG

Key terms

• Multi-Instance GPU: A feature that splits one GPU into isolated GPU instances, each with compute instances inside.

Try it

• MIG Partitioning: simulator lab

Sample question

On a DGX system you run 'sudo nvidia-smi -i 0 -mig 1' and get 'MIG mode is in pending enable state ... In use by another client'. What does NVIDIA's MIG guide say to do?