4.5 Storage performance

Key points

1. cuFile is the GDS API. Compatibility mode works without the direct path but is slower.

   What NVIDIA says (1)

   “In the /etc/cufile.json file, verify that allow_compat_mode is set to true . gdscheck -p displays whether the allow_compat_mode property is set to true .”

   — GPUDirect Storage Troubleshooting Guide

2. IOMMU (input-output memory management unit) translates device addresses. Here it forces a slower path.

   What NVIDIA says (2)

   “When the IOMMU setting is enabled, PCIe traffic will be routed through the CPU root ports.”

   — GPUDirect Storage Best Practices Guide

   “Before you install GDS, you must disable IOMMU.”

   — GPUDirect Storage Best Practices Guide

3. Fewer hops between GPU and NIC means higher throughput for GDS.

   What NVIDIA says (1)

   “The optimal path between a GPU and NIC will be one PCIe switch path designated as PIX. The least optimal path is designated as SYS”

   — GPUDirect Storage Configuration and Benchmarking Guide

4. Benchmark with several thread counts. The number of I/O threads strongly affects results.

   What NVIDIA says (1)

   “the number of processes/threads generating IO is critical to determining maximum performance. The gdsio tool provides for specifying random reads or random writes”

   — GPUDirect Storage Configuration and Benchmarking Guide

Key terms

• High-speed storage: Shared fast storage for all nodes, where datasets and checkpoints of running jobs belong.
• GPUDirect Storage: A direct data path between storage and GPU memory that avoids a CPU bounce buffer.
• IOMMU: The input-output memory management unit, which remaps device memory access and must be off for GDS.
• gdsio: The GDS benchmarking tool for storage read and write load.

Try it

• GPUDirect Storage: simulator lab
• Storage Bottleneck: simulator lab

Sample question

How do you check whether GDS is running in compatibility mode, where cuFile falls back to POSIX reads and writes?