Aegis-GPU2.2 AI data center architecture
The SuperPOD fabrics, storage, management network and Multi-Node NVLink.
Key points
A fabric is a network of switches and cables. Storage reads and writes would otherwise compete with GPU-to-GPU traffic.
What NVIDIA says (1)
“Storage traffic is dedicated to its own fabric to remove interference with the node-to-node application traffic that can degrade overall performance.”
Out-of-band (OOB) means separate from the data networks. It reaches the management controllers even when an OS is down.
What NVIDIA says (1)
“The out-of-band Ethernet network is used for system management using the BMC and provides connectivity to manage all networking equipment.”
HSS is shared, high-bandwidth storage for every node. Home directories use a separate NFS share.
What NVIDIA says (1)
“High-speed storage (HSS) provides shared storage to all nodes in the DGX SuperPOD. Store datasets, checkpoints, and other large files here.”
Users do not log in to GPU nodes directly. They work on login nodes, which are Slurm clients with the file systems mounted.
What NVIDIA says (1)
“Entry point to the DGX SuperPOD for users. CPU-based nodes that are Slurm clients with filesystems mounted to support development, job submission, job monitoring, and file management.”
NVLink connects GPUs directly at high speed. On GB200 and GB300, NVLink switches extend this across trays in a rack.
What NVIDIA says (1)
“Multi-Node NVLink is a capability enabled over an NVLink Switch network where multiple systems are interconnected to form a large GPU memory fabric also known as an NVLink Domain.”
Key terms
- Baseboard management controller: A small computer on the server board for remote power, monitoring and console access over the out-of-band network.
- DGX SuperPOD: NVIDIA's reference data center design of DGX systems, fabrics, storage and management nodes.
- Out-of-band network: A separate Ethernet network for BMC management, apart from job traffic.
- High-speed storage: Shared fast storage for all nodes, where datasets and checkpoints of running jobs belong.
- Multi-Node NVLink: NVLink extended across several systems through an NVLink Switch network.
Sample question
Why does the DGX SuperPOD put storage traffic on its own InfiniBand fabric?
Show the answer
Answer: To remove interference with node-to-node application traffic
A fabric is a network of switches and cables. Storage reads and writes would otherwise compete with GPU-to-GPU traffic.
What NVIDIA says (1)
“Storage traffic is dedicated to its own fabric to remove interference with the node-to-node application traffic that can degrade overall performance.”