2.5 Configuring MIG

Key points

1. MIG Manager is the GPU Operator component that applies MIG layouts. It watches the node label and reconfigures the GPUs.

   What NVIDIA says (2)

   “nvidia.com/mig.config=all-1g.10gb”

   — GPU Operator with MIG

   “GPU Operator deploys MIG Manager to manage MIG configuration on nodes in your Kubernetes cluster.”

   — GPU Operator with MIG

2. The strategy tells the Operator how MIG devices are exposed. You choose it at install time, before you configure MIG.

   What NVIDIA says (2)

   “single : MIG mode is enabled on all GPUs on a node.”

   — GPU Operator with MIG

   “mixed : MIG mode is not enabled on all GPUs on a node.”

   — GPU Operator with MIG

3. A GPU instance profile is a fixed slice size, such as 1g.5gb. -lgip lists profiles with free and total counts.

   What NVIDIA says (1)

   “$ nvidia-smi mig -lgip”

   — MIG User Guide: Getting Started with MIG

4. -cgi creates GPU instances. You can name a profile by ID (9) or by name (3g.20gb). -C also creates the matching compute instances.

   What NVIDIA says (2)

   “sudo nvidia-smi mig -cgi 9,3g.20gb -C”

   — MIG User Guide: Getting Started with MIG

   “Successfully created GPU instance ID 2 on GPU 0 using profile MIG 3g.20gb (ID 9)”

   — MIG User Guide: Getting Started with MIG

5. Each MIG device has its own UUID. CUDA_VISIBLE_DEVICES limits an app to the devices you name.

   What NVIDIA says (2)

   “$ nvidia-smi -L GPU 0: A100-SXM4-40GB”

   — MIG User Guide: Getting Started with MIG

   “$ CUDA_VISIBLE_DEVICES=MIG-c7384736-a75d-5afc-978f-d2f1294409fd ./BlackScholes &”

   — MIG User Guide: Getting Started with MIG

Key terms

• Multi-Instance GPU: A feature that splits one GPU into isolated GPU instances, each with its own memory and compute.

Try it

• MIG Partitioning: simulator lab

Sample question

With the GPU Operator, how do you apply the MIG profile all-1g.10gb to a node?