The big picture: one AI job from data to a served model

Every guide on this site covers one piece of the same story. A team has data. It trains a model on a cluster of GPUs. The GPUs and servers exchange results over a fast network. Software runs the whole thing. Operators keep it healthy. And finally the trained model is served so that applications can use it. Follow the six stages below. Each one links to the guides that explain it, and each guide opens by telling you where it sits in this picture.

The diagram is a picture of the list below. Use the list if you prefer text or are using a keyboard.

1. 1 Data goes in

   Raw data becomes training data

   AI work starts with data. NVIDIA's AI factory glossary says data pipelines are essential to LLMs because they turn raw, unstructured data into high-quality, structured tokens that models can learn from. Training passes over the same dataset many times, so how quickly storage and local caches can hand data to the GPUs matters.

   Read:

   • 1.7 The AI development and deployment life cycle: the data-preparation stage of the life cycle
   • 2.1 Hardware for training workloads: what a training server keeps in GPU memory, cache and storage
   • 2.5 Components of an accelerated cluster: shared storage and GPUDirect Storage

2. 2 GPUs train the model

   Repeat: predict, measure the error, adjust the weights

   Training is how a model learns: data passes through the network, the error is propagated back through its layers, and the model adjusts its weights and tries again. NVIDIA calls training 'a monster when it comes to consuming compute'. GPUs suit this work because a GPU runs thousands of threads in parallel, trading lower single-thread speed for much greater total throughput. Model weights have to fit in GPU memory, so a model that needs more than one GPU holds is split across GPUs.

   Read:

   • 1.3 AI, machine learning and deep learning: what a model, a neural network and a weight are
   • 1.2 Training vs inference: training compared with inference
   • 1.4 Why AI took off: why GPUs and transformers made this practical
   • 1.8 GPU vs CPU architecture: GPU compared with CPU
   • 2.1 Hardware for training workloads: GPU memory and what a training node provides

3. 3 The network moves data between GPUs

   Fast links inside a server and between servers

   GPUs in one job have to share results. All-reduce, for example, sums values such as gradients across all GPUs and gives every GPU the result. Inside a server, NVLink links GPUs directly. Between servers, NVIDIA's DGX SuperPOD uses NDR InfiniBand, and NCCL is the library that picks the paths. RDMA and GPUDirect RDMA let a network adapter move data into memory, including GPU memory, without the CPU copying it. A DPU can take network chores off the CPU.

   Read:

   • 2.9 High-speed network options: the high-speed link options
   • 2.8 Data center networking protocols: RDMA, GPUDirect RDMA, RoCE and subnet management
   • 2.10 DPUs in the data center: DPUs and BlueField
   • 2.7 Networking requirements for AI: what an AI cluster's network must provide

4. 4 Software runs it

   Driver, CUDA, libraries, containers, products

   Between the GPU and your AI application sit layers: the driver, CUDA, libraries, containers and serving software. NVIDIA packages many of them: the NGC catalog offers GPU-optimized containers, models and SDKs, and NVIDIA AI Enterprise is a supported software suite for production AI. The AI life cycle runs from data preparation to model development, deployment and monitoring, and TensorRT sits between training and deployment.

   Read:

   • 1.1 The NVIDIA software stack: the layers from driver to application
   • 1.6 NVIDIA solutions and what they are for: which NVIDIA product is for what
   • 1.7 The AI development and deployment life cycle: the life cycle from data to production

5. 5 Operations keep it healthy

   Schedule, share, watch, recover

   A GPU cluster is shared. Kubernetes, Slurm and NVIDIA Run:ai decide which job gets which GPUs. MIG, time-slicing and vGPU decide how one GPU is shared. DCGM measures health and utilization. And when something breaks, an Xid error written by the NVIDIA driver tells the operator what happened and what NVIDIA's catalog says to do.

   Read:

   • 3.2 Orchestration and job scheduling: orchestration and job scheduling
   • 3.4 Virtualizing GPUs: sharing one GPU safely
   • 3.3 Monitoring GPUs: the metrics worth watching
   • 3.1 Data center management and monitoring: Xid errors, health checks and recovery

6. 6 The model is served

   Inference: new requests get predictions

   A trained model makes predictions on new data. That is inference, and it is optimized for speed and efficiency. NVIDIA Triton Inference Server and NVIDIA NIM microservices run models behind an endpoint that applications call, and latency (time to answer one request) and throughput (work completed per unit of time) are the measures that matter.

   Read:

   • 1.2 Training vs inference: inference, latency and throughput
   • 1.6 NVIDIA solutions and what they are for: NIM and Triton
   • 1.5 AI use cases and industries: what applications use it for

Suggested reading order

Start at the top and read down. Each guide still shows its official NCA-AIIO objective number, so you can map your reading to the exam.

Prefer the exam outline? Browse the guides by exam domain and objective.