4.4 Troubleshooting Magnum IO

Key points

1. Magnum IO is NVIDIA's set of data-movement libraries, including NCCL and GPUDirect Storage. NCCL_DEBUG controls NCCL's debug output.

   What NVIDIA says (1)

   “The NCCL_DEBUG variable controls the debug information that is displayed from NCCL. This variable is commonly used for debugging.”

   — NCCL Environment Variables

2. It takes a list of name prefixes. A ^ excludes interfaces.

   What NVIDIA says (1)

   “The NCCL_SOCKET_IFNAME variable specifies which IP interfaces to use for communication.”

   — NCCL Environment Variables

3. Peer-to-peer (P2P) is a direct GPU-to-GPU path. Turning it off is a test, not a fix.

   What NVIDIA says (1)

   “The NCCL_P2P_DISABLE variable disables the peer to peer (P2P) transport, which uses CUDA direct access between GPUs, using NVLink or PCI.”

   — NCCL Environment Variables

4. GDS moves data straight between storage and GPU memory. ACS forces that traffic up through the CPU.

   What NVIDIA says (1)

   “For optimal GDS performance, disable ACS. Note To list all of the PCI switches that have ACS enabled, issue /usr/local/cuda/gds/tools/gdscheck -p .”

   — GPUDirect Storage Best Practices Guide

Key terms

• Magnum IO: NVIDIA's family of data-movement libraries, including NCCL and GPUDirect Storage.
• NCCL: The NVIDIA Collective Communications Library that moves data between GPUs for multi-GPU jobs.
• GPUDirect Storage: A direct data path between storage and GPU memory that avoids a CPU bounce buffer.
• Access Control Services: A PCIe feature that can force device-to-device traffic up through the CPU, slowing GDS.

Try it

• AllReduce Deep Dive: simulator lab
• GPUDirect Storage: simulator lab

Sample question

A distributed job hangs inside NCCL (the NVIDIA Collective Communications Library). What do you set first to see what NCCL is doing?