AI Operations

An Xid is an error report that the NVIDIA driver writes to the kernel log. Xid 48 is a double-bit ECC error. ECC (error-correcting code) memory can fix single-bit errors, but not double-bit ones, so this error is uncorrectable. NVIDIA's Xid Catalog lists the recovery for Xid 48 seen alone as RESET_GPU and the follow-up as RUN_FIELDDIAG (run field diagnostics). If Xid 63 or 64 appears with it, the catalog says to drain the node and reset.

Xid 79 means the GPU has fallen off the bus. The bus here is PCI Express (PCIe), the link between the GPU and the server's CPU. NVIDIA says the event is logged when the driver tries to reach the GPU over PCIe and finds it is not accessible. It is often caused by hardware failures on the PCIe link, so check system and kernel PCI event logs.

NVLink is NVIDIA's direct GPU-to-GPU link. An NVSwitch is a switch chip that connects many GPUs over NVLink. Xid 74 is logged when the GPU detects a problem with an NVLink connection to another GPU or an NVSwitch. The cause can be at the far end. If one GPU fails, a GPU linked to it may report Xid 74 simply because the link went down. For several error bits, NVIDIA's workflow says to check link mechanical connections, re-seat if needed, and run diagnostics if the issue persists.

Xid 13 is logged for general user application faults. NVIDIA says it is typically an out-of-bounds error, where code walks past the end of an array. Hardware failures can show up as Xid 13, but NVIDIA calls that rare. Because only one person's code triggers it, start by debugging that code.

Xid 45 is logged when an application aborts and the kernel driver tears down its work on the GPU. NVIDIA lists Ctrl-C, GPU resets and SIGKILL as examples. In many cases it is not a bug but a user or system action. If other Xids appear with it, follow the guidance for those.

Row remapping is how Ampere-generation and newer GPUs retire a faulty row of memory by mapping it to a spare row. Xid 63 records a row-remapping event, and the catalog's immediate action is IGNORE. Xid 64 records a row-remapping failure, and the catalog says RESET_GPU, then CONTACT_SUPPORT. An RMA (return merchandise authorization) is the return-for-replacement process. NVIDIA says the RMA criteria are met when the row-remapping failure flag is set and validated by field diagnostics.

Error containment is an Ampere-generation feature. It limits the impact of an uncorrectable ECC error to the application that hit it. For Xid 94, NVIDIA says the error is contained to one application, which must be restarted. All other applications are unaffected. NVIDIA still recommends resetting the GPU when convenient.

An uncontained error is one the GPU could not limit to a single application. For Xid 95, NVIDIA says the errors affect multiple applications and the GPU must be reset before applications can restart. So the operator drains the work from that GPU and resets it, instead of restarting just one service.

A device plugin is the Kubernetes mechanism that advertises special hardware to the scheduler. With NVIDIA's device plugin deployed, a container requests GPUs with the nvidia.com/gpu resource type. The scheduler then places the pod only on a node with that many free GPUs.

Orchestration is the automated placement and management of workloads on shared resources. NVIDIA Run:ai is a GPU orchestration and optimization platform that aims to maximize compute utilization for AI workloads. Administrators define projects, departments and user roles, and Run:ai enforces policies for fair resource distribution.

Gang scheduling, also called batch scheduling, treats a group of pods as one unit. The NVIDIA Run:ai Scheduler documentation describes it as a workload of many pods being either fully scheduled or fully pending. That stops half of a job from starting and sitting idle.

Base Command Manager (BCM) is NVIDIA's cluster management software. The DGX SuperPOD reference architecture says it automates provisioning and administration. NVIDIA's product page says it integrates with tools like Slurm or NVIDIA Run:ai. Slurm is a widely used job scheduler for high-performance computing.

Kubernetes is a container orchestration system. An Operator is a Kubernetes pattern for automating the management of a software component. NVIDIA says configuring GPU nodes by hand needs several components, such as drivers and container runtimes, and is error-prone. The GPU Operator automates all of them: the driver, the device plugin, the Container Toolkit, GPU Feature Discovery (GFD) labels and DCGM (Data Center GPU Manager) monitoring.

Field 311 counts double-bit volatile ECC errors. A double-bit error cannot be corrected by ECC, so any non-zero value is significant. NVIDIA's Xid Catalog logs uncorrectable errors as Xid 48, with RESET_GPU and RUN_FIELDDIAG as the actions when it appears alone.

When a GPU lowers its clocks to protect itself, the reason is recorded as a clock event. DCGM field 112 reports the current clock event reasons as a bitmask, so you can see whether heat, power or something else caused the drop. Field 100 shows the clock value, but not the reason.

DCGM (Data Center GPU Manager) exposes GPU telemetry as numbered fields. Field 310, DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, is the total of single-bit volatile ECC errors. Volatile means the count since the driver last loaded. Single-bit errors are corrected, so the trend matters more than any one value.

Prometheus is an open-source monitoring system that scrapes metrics over HTTP. DCGM Exporter exposes GPU metrics at an HTTP /metrics endpoint for tools such as Prometheus. Its default listen address is :9400.

DCGM (Data Center GPU Manager) is NVIDIA's suite for managing and monitoring data center GPUs. NVIDIA says it includes active health monitoring, diagnostics, system alerts and governance policies such as power and clock management. Teams can use it alone or integrate it into cluster management and monitoring tools.

A streaming multiprocessor (SM) is one of the many processing blocks inside a GPU. DCGM defines graphics engine activity as the fraction of time any portion of the graphics or compute engines were active. So it can be high while most SMs are idle. DCGM's SM activity metric is the fraction of time at least one warp was active, averaged over all SMs. A warp is a group of threads the SM runs together. That metric shows how much of the chip is really working.

NVIDIA says MIG lets supported GPUs be securely partitioned into up to seven separate GPU instances. MIG mode is not enabled by default. An administrator enables it, then creates instances from profiles such as 1g.10gb on H100. A profile name gives the instance's share of compute (1g = one compute slice) and its memory size (10gb).

A virtual machine (VM) is a software-defined computer that runs on a hypervisor. NVIDIA vGPU software is installed on a physical GPU in a server and creates virtual GPUs that can be shared across multiple VMs. NVIDIA says this raises GPU utilization while keeping management and security centralized.

Multi-Instance GPU (MIG) partitions one physical GPU into separate GPU instances. NVIDIA says each instance gets dedicated compute and memory resources. Each instance's processors have isolated paths through the whole memory system. MIG also provides fault isolation between clients such as VMs, containers or processes.

Time-slicing lets workloads on an oversubscribed GPU take turns. Oversubscribed means more workloads than GPUs. NVIDIA says that, unlike MIG, there is no memory or fault isolation between the shared replicas. Time-slicing trades MIG's isolation for the ability to share a GPU among more users.