5.5 Storage optimization

Key points

1. ACS forces peer-to-peer PCIe traffic up through the root complex. GDS then cannot bypass the CPU. NVIDIA says to disable ACS for best GDS performance. gdscheck -p lists the switches that have it on.

   What NVIDIA says (2)

   “For optimal GDS performance, disable ACS.”

   — GPUDirect Storage Best Practices Guide

   “To list all of the PCI switches that have ACS enabled, issue /usr/local/cuda/gds/tools/gdscheck -p .”

   — GPUDirect Storage Best Practices Guide

2. P2P DMA (peer-to-peer direct memory access) lets a NIC write straight into GPU memory. It works best when the NIC and GPU share a PCIe switch. Crossing sockets adds a slow hop.

   What NVIDIA says (2)

   “For the P2P DMA to function efficiently, NICs, NVMes and GPUs should be under a PCIe switch when possible.”

   — GPUDirect Storage Best Practices Guide

   “ensure at least one NIC is in the same CPU socket as the GPU.”

   — GPUDirect Storage Best Practices Guide

3. cuFile is the GDS library API. Its settings live in /etc/cufile.json. GDS splits a large read or write into chunks of max_direct_io_size. Bigger chunks mean fewer calls.

   What NVIDIA says (2)

   “For the requested IO size, GDS issues IO requests sequentially in chunks of reads/writes based on the max_direct_io_size parameter.”

   — GPUDirect Storage Best Practices Guide

   “Larger values of max_direct_io_size will result in a reduced number of calls to the IO stack”

   — GPUDirect Storage Best Practices Guide

Key terms

• GPUDirect Storage: A direct data path between storage and GPU memory that bypasses a CPU bounce buffer.
• Access Control Services: A PCIe feature that can force peer-to-peer traffic up to the CPU, which slows GDS and GPUDirect.
• NUMA: Non-uniform memory access: each CPU socket has its own close memory, so processes should run near their GPU and NIC.

Sample question

GDS throughput is low on a server with PCIe switches. Which NVIDIA best practice applies?