Aegis-GPU3.4 GPU and DOCA drivers
Installing, checking, upgrading and removing the NVIDIA driver and DOCA-Host.
Key points
The driver includes kernel modules, which are code that runs inside the Linux kernel. They are built against kernel headers. If the headers do not match the running kernel, the build fails. So install the headers for uname -r before the driver.
What NVIDIA says (2)
“it is best to manually ensure the correct version of the kernel headers and development packages are installed prior to installing the NVIDIA driver”
“# apt install linux-headers-$(uname -r)”
The open kernel modules are NVIDIA's open-source GPU kernel driver. After enabling NVIDIA's repository, the guide installs them with apt install nvidia-open. For a compute-only server, it can install libnvidia-compute and nvidia-dkms-open instead.
What NVIDIA says (2)
“# apt install nvidia-open”
“apt -V install libnvidia-compute nvidia-dkms-open”
Installed is not the same as loaded. The kernel reports the loaded driver in /proc. The guide reads /proc/driver/nvidia/version.
What NVIDIA says (1)
“When the driver is loaded, the driver version can be found by executing the following command: $ cat /proc/driver/nvidia/version”
APT is Ubuntu's package manager. Pinning tells APT to stay on one driver branch or version. NVIDIA says the upgrade command stays the same. The pinning configuration decides where it lands.
What NVIDIA says (2)
“When upgrading the driver, whether configured to a pinned branch or the latest available, the command to execute is always the same; what matters is the APT pinning configuration:”
“# apt dist-upgrade”
DOCA-Host is NVIDIA's host software for ConnectX and BlueField: drivers, libraries and tools. OFED is the older InfiniBand and RDMA driver stack it replaced. Upgrading from 2.5.x needs a full uninstall first. From 2.6.0 or later, apt can upgrade in place.
What NVIDIA says (2)
“To upgrade from DOCA version 2.5.x, all DOCA and OFED related packages should be removed.”
“Full uninstallation is required before installing DOCA-Host:”
openibd is the service that loads the InfiniBand and RDMA drivers. MST (Mellanox Software Tools) creates device files used by the firmware tools. The DOCA guide updates firmware, loads the drivers, and initializes MST.
What NVIDIA says (4)
“host# sudo apt install -y doca-all”
“Update the firmware: host# sudo apt install -y mlnx-fw-updater”
“Load the drivers: host# sudo /etc/init.d/openibd restart”
“Initialize MST: host# sudo mst restart”
Removing packages with apt keeps the package database correct. Deleting files by hand does not. The guide purges a list of driver packages. The list includes nvidia-fabricmanager and nvidia-persistenced.
What NVIDIA says (2)
“Follow the below steps to properly uninstall the NVIDIA driver from your system.”
“# apt remove --autoremove --purge -V”
Key terms
- DOCA: NVIDIA's software framework and driver stack for BlueField and ConnectX; DOCA-Host installs on the host.
Try it
Sample question
Before installing the NVIDIA data center driver on Ubuntu, what does NVIDIA say to check first?
Show the answer
Answer: That kernel headers and development packages match the running kernel (uname -r).
The driver includes kernel modules, which are code that runs inside the Linux kernel. They are built against kernel headers. If the headers do not match the running kernel, the build fails. So install the headers for uname -r before the driver.
What NVIDIA says (2)
“it is best to manually ensure the correct version of the kernel headers and development packages are installed prior to installing the NVIDIA driver”
“# apt install linux-headers-$(uname -r)”
Practice 3.4 (7 questions) Full Control Plane Installation and Configuration guide
← 3.3 Cluster setup: categories, Slurm, Enroot, Pyxis · 3.5 NVIDIA Container Toolkit →