Aegis-GPUNCA-AIIO glossary
The official terms you will meet on the exam and in the field. Each has a one-sentence plain definition and the NVIDIA quote it is based on.
A
- Accelerated computing
Using specialized hardware such as GPUs to speed up demanding work through parallel processing.
What NVIDIA says (2)
“A GPU provides much higher instruction throughput and memory bandwidth than a CPU within a similar price and power envelope.”
“Accelerated computing is the use of specialized hardware to dramatically speed up work, using parallel processing that bundles frequently occurring tasks.”
- Aisle containment
Enclosing the hot or cold aisle so hot exhaust air cannot mix back into server inlets.
What NVIDIA says (1)
“In either case, the main benefit of aisle containment is the prevention of air recirculation from the hot aisle to the cold aisle”
- Artificial intelligence
Software that performs tasks that normally need human intelligence; machine learning and deep learning are subsets of it.
What NVIDIA says (1)
“As a subset of AI, machine learning in its most elemental form uses algorithms to parse data, learn from it, and then make predictions or determinations about something in the real world.”
B
- Bandwidth
How much data a link or memory can move per second, for example in GB/s.
What NVIDIA says (1)
“640 GB of aggregated HBM3 memory with 24 TB/s of aggregate memory bandwidth”
- Batch
The group of samples a model processes in one step; batch size is how many samples are in that group.
What NVIDIA says (1)
“Mixed precision models use less memory than FP32, so it is possible to increase the batch size when running with AMP.”
C
- CapEx and OpEx
CapEx is money spent up front on assets; OpEx is ongoing spending such as a cloud bill.
What NVIDIA says (1)
“Cloud-based solutions offer a cost-effective way to start AI initiatives by reducing acquisition costs and shifting capital expenditures (CapEx) to operational expenditures (OpEx).”
- Central processing unit
The general-purpose processor in a server: a few fast cores with large caches, built to run a few threads quickly.
What NVIDIA says (3)
“GPUs are specialized for highly parallel computations and devote more transistors to data processing units, while CPUs dedicate more transistors to data caching and flow control.”
“a CPU is designed to excel at executing a serial sequence of operations (called a thread) as fast as possible and can execute a few tens of these threads in parallel”
“Architecturally, the CPU is composed of just a few cores with lots of cache memory that can handle a few software threads at a time.”
- Checkpoint
A saved copy of training state so a job can resume after a failure.
What NVIDIA says (1)
“As DL models grow in size and time-to-train, writing checkpoints is necessary for fault tolerance.”
- Container
A package of an application together with its software dependencies (libraries, configuration, tools) that runs the same on any host.
What NVIDIA says (1)
“All of them also include the needed GPU libraries, configuration files, and tools to rebuild the container.”
- CUDA
NVIDIA's parallel computing platform; CUDA-X libraries built on it accelerate AI and HPC.
What NVIDIA says (3)
“to enable any computational workload to use the throughput capability of GPUs independent of graphics APIs.”
“Libraries like cuBLAS, cuFFT, cuDNN, and CUTLASS are just a few examples of libraries that help developers avoid reimplementing well-established algorithms.”
“NVIDIA CUDA-X™, built on CUDA, is a collection of libraries that deliver dramatically higher performance across application domains, including AI and HPC.”
- cuDNN
A GPU library of tuned building blocks, such as convolutions, for deep neural networks.
What NVIDIA says (1)
“The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks.”
D
- Data processing unit
A system on a chip with CPU cores, a fast network interface and acceleration engines that moves and processes data center traffic.
What NVIDIA says (2)
“BlueField-3 DPUs offload, accelerate, and isolate software-defined networking, storage, security, and management functions, significantly enhancing data center performance, efficiency, and security.”
“Specialists in moving data in data centers, DPUs, or data processing units, are a new class of programmable processor and will join CPUs and GPUs as one of the three pillars of computing.”
- DCGM
NVIDIA tooling for GPU health monitoring, diagnostics, alerts and policies across a cluster.
What NVIDIA says (3)
“Active Health Checks (GPU subsystems)”
“GPU Diagnostics (Diagnostic Levels - 1, 2, 3)”
“It includes active health monitoring, comprehensive diagnostics, system alerts, and governance policies including power and clock management.”
- DCGM Exporter
A service that publishes DCGM GPU metrics on an HTTP /metrics endpoint for Prometheus.
What NVIDIA says (1)
“DCGM Exporter is written in Go and exposes GPU metrics at an HTTP endpoint ( /metrics ) for monitoring solutions such as Prometheus.”
- Deep learning
A subset of machine learning that uses neural networks with many layers to learn patterns directly from data such as images or text.
What NVIDIA says (1)
“Deep learning is a subset of machine learning, with the difference that DL algorithms can automatically learn representations from data such as images, video, or text, without introducing human domain knowledge.”
- DGX SuperPOD
NVIDIA's reference design for an AI cluster made of DGX systems, InfiniBand and Ethernet networking, management nodes and storage.
What NVIDIA says (1)
“The DGX SuperPOD architecture is a combination of DGX systems, InfiniBand and Ethernet networking, management nodes, and storage.”
- DGX system
NVIDIA's integrated GPU server; a DGX H100 system has eight GPUs.
What NVIDIA says (1)
“The DGX H100 system, which is the fourth-generation NVIDIA DGX system, delivers AI excellence in an eight GPU configuration.”
- DOCA
The software framework (SDK and runtime) for building services on BlueField DPUs and SuperNICs.
What NVIDIA says (1)
“DOCA contains a runtime and development environment, including libraries and drivers for device management and programmability”
E
- ECC
Memory protection that fixes single-bit errors and detects double-bit errors.
What NVIDIA says (1)
“DCGM_FI_DEV_ECC_DBE_VOL_TOTAL 311 Total double bit volatile ECC errors.”
F
- Fabric
The network that connects the servers: the switches, cables and adapters together.
What NVIDIA says (1)
“An InfiniBand fabric is composed of switches and channel adapter (HCA/TCA) devices.”
G
- Generative AI
AI models that learn patterns in existing data and use them to create new content such as text, images or code.
What NVIDIA says (1)
“Generative AI models use neural networks to identify the patterns and structures within existing data to generate new and original content.”
- GPU Operator
A Kubernetes operator that installs and manages the NVIDIA driver, device plugin, container toolkit and monitoring.
What NVIDIA says (1)
“The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPU.”
- GPUDirect RDMA
A direct path between GPU memory and a peer device such as a network adapter over PCI Express.
What NVIDIA says (1)
“GPUDirect RDMA is a technology introduced in Kepler-class GPUs and CUDA 5.0 that enables a direct path for data exchange between the GPU and a third-party peer device using standard features of PCI Express.”
- GPUDirect Storage
A direct path that moves data between storage and GPU memory without a copy through CPU memory.
What NVIDIA says (1)
“GPUDirect® Storage (GDS) enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU.”
- Gradient
The correction signal computed in training that says how much, and in which direction, to adjust each weight.
What NVIDIA says (1)
“Distributed Data Parallelism (DDP) keeps the model copies consistent by synchronizing parameter gradients across data-parallel GPUs before each parameter update.”
- Graphics processing unit
A processor with hundreds or thousands of cores that runs many threads in parallel for high total throughput.
What NVIDIA says (2)
“a GPU is designed to excel at executing thousands of threads in parallel, trading off lower single-thread performance to achieve much greater total throughput.”
“In contrast, a GPU is composed of hundreds of cores that can handle thousands of threads simultaneously.”
I
- Inference
Using a trained model to make predictions or generate outputs on new data.
What NVIDIA says (1)
“Inference is the process where a trained AI model generates new outputs by reasoning and making predictions on new data — classifying inputs and applying learned knowledge in real time.”
- InfiniBand
A high-performance, low-latency, RDMA-capable network used to connect GPU servers and storage.
What NVIDIA says (1)
“InfiniBand is a high-performance, low latency, RDMA capable networking technology”
K
- Kubernetes
An open-source system that orchestrates containers: it deploys, scales and manages them across a cluster.
What NVIDIA says (1)
“Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.”
L
- Large language model
A very large neural network trained on huge text datasets that can generate human-like text.
What NVIDIA says (1)
“However, large language models, which are trained on internet-scale datasets with hundreds of billions of parameters, have now unlocked an AI model’s ability to generate human-like content.”
- Latency
How long one request or one transfer takes from start to finish.
What NVIDIA says (1)
“TensorRT includes inference compilers, runtimes, and model optimizations that deliver low latency and high throughput for production applications.”
- Lossless Ethernet
Ethernet configured to pause traffic instead of dropping packets when buffers fill, which RoCE needs.
What NVIDIA says (2)
“RDMA over Converged Ethernet (RoCE) is a mechanism to provide this efficient data transfer with very low latencies on lossless Ethernet networks.”
“For example, PFC can provide lossless service for the RoCE traffic and best-effort service for the standard Ethernet traffic.”
M
- Machine learning
A subset of AI where algorithms learn from data to make predictions instead of following hand-written rules.
What NVIDIA says (1)
“As a subset of AI, machine learning in its most elemental form uses algorithms to parse data, learn from it, and then make predictions or determinations about something in the real world.”
- Mixed precision
Training with lower-precision number formats where safe, which saves memory and runs math faster on Tensor Cores.
What NVIDIA says (1)
“Third, math operations run much faster in reduced precision, especially on GPUs with Tensor Core support for that precision.”
- MLOps
Practices for building, deploying and maintaining ML models in production, extending DevOps.
What NVIDIA says (1)
“short for machine learning operations, is a set of practices and principles that aims to streamline the development, deployment, and maintenance of machine learning (ML) models in production environments.”
- Multi-Instance GPU
Splitting one GPU into up to seven isolated instances, each with its own compute and memory.
What NVIDIA says (1)
“allows GPUs (starting with NVIDIA Ampere architecture) to be securely partitioned into up to seven separate GPU Instances for CUDA applications”
N
- N+1 redundancy
Providing the N power sources you need plus one spare, so one can fail without an outage.
What NVIDIA says (1)
“Due to this requirement, the data center must minimally provide N+1 power, where N equals two power sources.”
- NCCL
The library that moves data between GPUs inside and across nodes, with operations such as all-reduce.
What NVIDIA says (2)
“is a library providing inter-GPU communication primitives that are topology-aware and can be easily integrated into applications.”
“NCCL implements both collective communication and point-to-point send/receive primitives.”
- NGC catalog
NVIDIA's catalog of GPU-optimized containers, pretrained models and SDKs.
What NVIDIA says (1)
“The NGC Catalog consists of containers, pretrained models, Helm charts for Kubernetes deployments, and industry-specific AI toolkits with software development kits (SDKs).”
- NVIDIA NIM
Prebuilt, optimized inference microservices for deploying AI models on NVIDIA GPUs anywhere.
What NVIDIA says (2)
“NVIDIA NIM microservices are a set of easy-to-use microservices for accelerating the deployment of foundation models on any cloud or data center”
“NVIDIA NIM™ provides prebuilt, optimized inference microservices for rapidly deploying the latest AI models on any NVIDIA-accelerated infrastructure—cloud, data center, workstation, and edge.”
- NVLink
NVIDIA's direct GPU-to-GPU interconnect inside a server.
What NVIDIA says (1)
“is a direct GPU-to-GPU interconnect that scales multi-GPU input/output (IO) in the server.”
- NVLink Switch
A switch chip that connects many NVLinks so all GPUs can talk to each other at full NVLink speed.
What NVIDIA says (2)
“which connects multiple NVLinks to provide all-to-all GPU communication at the total NVLink speed.”
“The NVIDIA NVLink Switch chips connect multiple NVLinks to provide all-to-all GPU communication at full NVLink speed across the entire rack.”
O
- Oversubscription
When more bandwidth (or demand) can enter a layer than its capacity can serve.
What NVIDIA says (1)
“When there is a delta between the capacity of a resource in the data center and the demand for that resource, the resource is said to be oversubscribed.”
P
- PCI Express
The standard interconnect inside a server that links the CPU to GPUs, network adapters and storage.
What NVIDIA says (2)
“It supports a variety of interconnect technologies including PCIe, NVLINK, InfiniBand Verbs, and IP sockets.”
“These routines are optimized to achieve high bandwidth and low latency over PCIe,NVIDIA NVLink™, and other high-speed interconnects within a node and over NVIDIA networking across nodes.”
- Pretrained model
A model already trained on a large dataset that you can use as is or fine-tune for your task.
What NVIDIA says (1)
“A pretrained AI model is a deep learning model that’s trained on large datasets to accomplish a specific task, and it can be used as is or customized to suit application requirements across multiple industries.”
- Prometheus
An open-source monitoring system that collects metrics, such as DCGM Exporter GPU metrics, from /metrics endpoints.
What NVIDIA says (1)
“DCGM Exporter is written in Go and exposes GPU metrics at an HTTP endpoint ( /metrics ) for monitoring solutions such as Prometheus.”
R
- Rack density
How many systems are placed in one rack; it is limited by the power and cooling available per rack.
What NVIDIA says (1)
“However, rack densities can be customized to fit within the available power and cooling capacities at the data center.”
- Rail-optimized network
A compute fabric where each same-numbered port on every node connects to its own set of leaf switches, keeping that traffic one hop apart.
What NVIDIA says (1)
“Traffic per rail of the DGX H100 systems is always one hop away from the other 31 nodes in a SU.”
- RDMA over Converged Ethernet
RDMA carried over Ethernet networks instead of InfiniBand.
What NVIDIA says (1)
“RDMA over Converged Ethernet (RoCE) is a mechanism to provide this efficient data transfer with very low latencies on lossless Ethernet networks.”
- Remote direct memory access
Moving data straight into another machine's memory without the CPU copying it, which cuts latency and CPU load.
What NVIDIA says (1)
“Storage is provided over InfiniBand and leverages RDMA to provide maximum performance and minimize CPU overhead.”
S
- Scalable unit
A repeatable building block of 32 DGX H100 systems used to grow a DGX SuperPOD.
What NVIDIA says (1)
“The system is built upon building blocks of scalable units (SU), each containing 32 DGX H100 systems, which provides for rapid deployment of systems of multiple sizes.”
- SHARP
Switch-based in-network computing that performs collective operations so less data crosses the network.
What NVIDIA says (1)
“technology improves the performance of MPI and Machine Learning collective operation, by offloading collective operations from CPUs and GPUs to the network”
- Slurm
An open-source cluster scheduler that queues jobs and assigns them nodes and GPUs.
What NVIDIA says (1)
“Slurm is an open-source cluster resource management and job scheduling system”
- Spectrum-X Ethernet
NVIDIA's AI-optimized Ethernet platform of switches and SuperNICs.
What NVIDIA says (3)
“NVIDIA Spectrum-X is an AI-optimized Ethernet networking platform that combines NVIDIA Spectrum switches with the BlueField-3 SuperNIC”
“to deliver high-bandwidth, lossless RoCE for the GPU-to-GPU compute (east-west) network.”
“It combines purpose-built Ethernet switches and SuperNICs to deliver high effective bandwidth, low latency, and strong performance isolation for hyperscale AI infrastructure”
- Streaming multiprocessor
The GPU compute unit that runs threads; a GPU has many SMs, and MIG divides them between instances.
What NVIDIA says (1)
“available GPU compute resources (including streaming multiprocessors or SMs, and GPU engines such as copy engines or decoders)”
- Subnet Manager
The service that configures an InfiniBand fabric; one must be running at all times.
What NVIDIA says (1)
“All InfiniBand-compliant ULPs require a proper operation of a Subnet Manager (SM) running on the InfiniBand fabric, at all times.”
T
- Tensor Core
A specialized unit inside NVIDIA GPUs that speeds up the matrix math used in deep learning.
What NVIDIA says (1)
“Tensor Cores were introduced in the NVIDIA Volta™ GPU architecture to accelerate matrix multiply and accumulate operations for machine learning and scientific applications.”
- Throughput
How much total work finishes per second, for example requests or samples per second.
What NVIDIA says (1)
“they allow enterprises to optimize throughput and latency so they can meet their service level agreements across a variety of use cases.”
- Time-slicing
Sharing a GPU by letting workloads take turns, without the memory and fault isolation of MIG.
What NVIDIA says (1)
“Time-slicing trades the memory and fault-isolation that is provided by MIG for the ability to share a GPU by a larger number of users.”
- Total cost of ownership
The full cost of a system over its life, including compute, storage and maintenance.
What NVIDIA says (1)
“IT leaders should evaluate the total cost of ownership (TCO) over time and consider factors such as data storage, compute resources, and ongoing maintenance.”
- Training
The process of feeding data through a model and adjusting its weights until its outputs are accurate.
What NVIDIA says (1)
“Instead, the error is propagated back through the network’s layers, and the model must adjust its weights and try again.”
- Transformer
A neural network design that uses attention to relate every part of an input to every other part, and runs well in parallel.
What NVIDIA says (1)
“Transformer models apply an evolving set of mathematical techniques, called attention or self-attention, to detect subtle ways even distant data elements in a series influence and depend on each other.”
- Triton Inference Server
Open-source software that serves models from many frameworks for inference.
What NVIDIA says (1)
“Triton Inference Server is an open source inference serving software that streamlines AI inferencing.”
V
- Virtual GPU
NVIDIA software that creates virtual GPUs shared across virtual machines.
What NVIDIA says (1)
“NVIDIA virtual GPU software enables multiple virtual machines (VMs) to have simultaneous, direct access to a single physical GPU”
X
- Xid error
An error report the NVIDIA driver writes to the kernel log, identified by a number such as 48 or 79.
What NVIDIA says (1)
“This event is logged when the GPU detects that an uncorrectable error occurs on the GPU.”