Inside an 8-GPU HGX Server: A Beginner's Hardware Guide

Assumes you know nothing. Every term is explained before it's used — including what kind of thing it is.

How to read this guide

Technical documentation constantly mixes together four or five completely different categories of thing without ever telling you which is which. That's the main reason it feels impenetrable. So in this guide, every new term gets a tag:

• PART — a physical object you could hold in your hand
• STANDARD — an agreement or set of rules. Not an object. You can't hold it. It's a specification everyone follows so equipment from different companies works together.
• SOFTWARE — instructions, stored as files, that run on a processor
• MEASURE — a unit or number describing something
• ROLE — a job or a person, not a thing

And every component gets answered in this order: why does it need to exist → what problem does it solve → what it actually is → the details.

Part 0: What is this machine, and why does it exist?

Start with the problem

Someone wants to train an artificial intelligence model. Training means: show the computer billions of examples, and after each one, adjust billions of internal numbers slightly. The math is simple — mostly multiplication and addition — but there is a staggering amount of it.

An ordinary computer's processor does this arithmetic one small batch at a time, very quickly. Even so, "very quickly" times "an unimaginable amount" equals centuries. That's the problem.

Why a different kind of processor solves it

A CPU (Central Processing Unit) PART — a physical chip, roughly the size of a large postage stamp — is a general-purpose processor. It's designed to handle any instruction you throw at it, making complicated decisions, running the operating system, deciding what happens next. To do that flexibly, it dedicates most of its silicon to decision-making machinery, leaving relatively few arithmetic units. A modern server CPU might do a few dozen calculations at once.

A GPU (Graphics Processing Unit) PART — also a physical chip, but much larger and hotter — made the opposite trade. It was originally built for video games: to draw a screen you must compute the color of two million pixels, and each pixel's math is independent of every other pixel's. So GPU designers stripped out most of the decision-making machinery and filled the chip with thousands of simple arithmetic units instead.

That makes a GPU nearly useless for general computing and extraordinary at one specific thing: doing the same simple math on enormous numbers of values simultaneously.

Which, it turned out, is exactly what AI training is.

Rough analogy: a CPU is one brilliant mathematician who can solve any problem you describe. A GPU is ten thousand schoolchildren who can only do multiplication tables — but do them all at once. For AI work, the schoolchildren win by a margin so large it changed the industry.

So what is this class of server?

An 8-GPU HGX-class server is a computer built around a single design goal: hold eight of these enormous GPU chips, and keep all eight supplied with data, electricity, and cold air at all times.

Every other component in this machine exists in service of those eight. It's a shape of server built by several different manufacturers around the same underlying NVIDIA GPU baseboard design, so the exact model names and firmware tool names differ from vendor to vendor — but the architecture underneath is close enough that this guide's structure carries over. Once you know the shape, translating it to your specific manufacturer's manual is mostly a vocabulary exercise.

Basic facts common to this class of machine:

• It's a rack server PART — a flat, wide metal box that slides into a tall metal frame called a rack, like a drawer in a filing cabinet. Data centers are rooms full of racks.
• It's typically 6U tall. U MEASURE is the standard height unit for rack equipment: 1U = 1.75 inches. So 6U ≈ 10.5 inches. Most servers are 1U or 2U — 8-GPU machines are unusually tall because the GPUs and the cooling hardware they need don't fit in a thin box.

Model naming and firmware-tool names vary by manufacturer, so treat any specific labels in your own server's documentation as the authority — this guide teaches the shared architecture underneath them.

Part 1: The floor plan — a three-story building

Before any details, get this picture in your head. It makes everything afterward readable.

Service documentation for these machines constantly says "Layer 2" and "Layer 3." Those aren't abstract concepts — they mean physical height levels inside the box. Stacked floors.

Cables travel between floors through cutouts — literally holes cut in the sheet metal so a cable can pass from one level to the next. When a manual says "route through the chassis cutout," it means "thread this cable through the hole to the floor below."

That's it. That's the whole model. Hold onto it.

Part 2: Power — following the electricity

Why this is a bigger deal here than in a normal computer

A desktop computer draws maybe 300 watts. This machine draws thousands. Eight GPUs at up to 700 watts each is 5,600 watts from the GPUs alone, before you count CPUs, memory, drives, and fans.

At that scale, ordinary approaches stop working. Wires that would carry a desktop's power would glow red here. So the power system is built in deliberate stages, each solving a specific problem.

Stage 1 — power supplies: converting wall power into chip power

Why needed: the electricity in the wall is AC (alternating current — it reverses direction 50 or 60 times per second). Computer chips need DC (direct current — steady, one direction). Something must convert one to the other.

A PSU (Power Supply Unit) PART is a metal brick that does exactly that conversion. Machines in this class typically carry six of them at the rear, each rated around 2800W (some configurations use 3200W).

Why six? Two reasons stacked together. First, raw capacity — one PSU couldn't supply this machine. Second, redundancy: if one fails, the others carry the load and the server keeps running. In a data center, unplanned downtime costs more than spare hardware.

Stage 2 — PDB: the distribution point

Why needed: you now have six PSUs producing power and dozens of components needing it. Wiring every PSU directly to every component would be an unmanageable tangle.

A PDB (Power Distribution Board) PART is a circuit board that solves this: all PSUs feed into it, and it fans power out to everything else. It's the breaker panel of the house.

It also matters for repairs: the PDB is the hub that every cable disconnects from when you tear the machine down. If you're doing major service, the PDB is where the cable marathon happens.

Stage 3 — PBB: the high-current stage

Why needed: GPUs need far more current than ordinary components, and current at that magnitude has to be handled differently.

The PBB (Power Baseboard) PART is a large board that takes power from the PDB and delivers it to the GPUs and switches:

• 100A at 12 volts → to slots and switches
• 150A at 54 volts → to the GPU baseboard

Why 54 volts instead of 12? Basic electrical physics: for a given amount of power, higher voltage means lower current, and lower current means less heat lost in the wiring. Raising the voltage is how you deliver enormous power without melting the delivery path.

150 amps is still enormous. This is why the machine uses bus bars PART — solid copper strips instead of wires. A wire thick enough to carry 150A safely would be too stiff and bulky to route; a flat copper bar handles it cleanly.

Stage 4 — the GPUs

Up to 700W each for H100/H200/H800-class GPUs; 500W for A100-class. Eight of them. Now the six power supplies make sense.

Power-capping failover — why "just shut down" is the wrong answer

Why needed: imagine a PSU fails 40 hours into a training run. A normal server's response — shut down to protect itself — destroys 40 hours of work.

A smarter response is possible: instead of shutting down, the firmware deliberately slows the GPUs so total power draw fits within the surviving PSUs. The job continues, slower. Like a car dropping to a lower gear on a steep hill instead of stalling.

The catch worth memorizing: this graceful-throttling behavior typically exists only on the NVIDIA-GPU version of this server class. AMD and Intel accelerator variants generally don't have it. This is a favorite exam question.

Part 3: Data — why the CPU talks to the GPUs at all

The relationship: GPUs can't act on their own

This is the "why" that gets skipped everywhere, so let's do it properly.

A GPU is not an independent computer. It has no operating system. It doesn't boot. It can't open a file, receive a network packet, or decide what to do next. It sits idle until something tells it: here is a block of data, here is the calculation to perform on it, go.

That "something" is the CPU. The CPU runs the operating system and the actual program. It is the only component that can make decisions.

So a training job physically looks like this:

1. The training data lives on storage drives (or arrives over the network).
2. The CPU reads that data in — it's the only part that can talk to drives and network cards.
3. The CPU pushes a chunk of that data across to a GPU's own memory.
4. The CPU tells the GPU which calculation to run.
5. The GPU runs it at massive parallel speed.
6. The CPU collects the results and pushes the next chunk.
7. Repeat, billions of times.

Therefore: the CPU must have a fast connection to every GPU (to feed them), to the storage drives (to read training data), and to the network cards (to receive data and coordinate with other servers). The CPU is the hub. Everything routes through it.

And now the real problem becomes visible: a CPU chip has a physically limited number of connection points. Only so many wires can leave a piece of silicon. But this machine has eight GPUs, multiple network cards, and up to sixteen drives all needing a fast path to that one CPU.

There aren't enough connections to go around. Something has to solve that.

The solution: a switch

A switch PART is a chip that takes one incoming connection and splits it among many devices, routing traffic to whichever destination needs it at that moment.

The logic is the same as an office with eight employees and two phone lines. You don't need eight lines, because all eight employees are rarely on the phone simultaneously. A switchboard shares the two lines among them, connecting whoever's actually talking. You trade a little peak capacity for a huge saving in wiring.

PCIe — what it actually is (and isn't)

Before the switch details: PCIe (Peripheral Component Interconnect Express) STANDARD.

This is the part that confuses everyone, so let's be precise: PCIe is not a device. It's not a chip or a card. It is a published rulebook — a specification document maintained by an industry group — that defines how components inside a computer send data to each other. It specifies the electrical signals, the connector shapes, the message format, the speeds.

Why does a rulebook matter? Because a GPU made by one company has to work in a server made by another using a switch chip made by a third. That only happens if all three companies build to the same agreed rules. PCIe is that agreement.

So when someone says "the GPU connects over PCIe," they mean: there are physical copper traces and a physical connector between the GPU and the rest of the machine, and both ends send signals according to the PCIe rulebook. The wires are physical. PCIe is the rules the wires obey.

You'll see the same word used three ways, and context tells you which:

• "a PCIe slot" → PART the physical connector on a board
• "a PCIe device" → PART any card that follows the standard
• "PCIe Gen4" → STANDARD a specific version of the rulebook

The standard defines two things you'll be checking constantly:

Width — how many parallel lanes of wiring the connection uses: x1, x4, x8, x16. Think highway lanes. More lanes, more data at once. x16 is the maximum and what GPUs use.

Generation — which version of the rulebook, each faster than the last. Gen4 runs at 16 GT/s — "gigatransfers per second" MEASURE, roughly meaning 16 billion signal transitions per second per lane. Remember 16 GT/s and x16; you'll verify both with a command in Part 6.

PSB — the PCIe Switch Board

Now it assembles. A PSB (PCIe Switch Board) PART is a removable circuit board holding a PCIe switch chip. It's the switchboard solving the "not enough CPU connections" problem.

Machines in this class typically carry four PSBs, numbered PSB 1 through PSB 4, left to right.

Each PSB provides:

• 2 × x16 uplinks to the CPU — the shared "phone lines" back to the decision-maker
• 2 × x16 to riser cards — where network cards plug in
• 2 × x16 to the GPUs (via the PBB)
• 2 × x16 fabric/neighbor links — so PSBs can pass traffic to each other directly

An asymmetry to memorize: PSB 1 and PSB 4 have one link to the system board CPU. PSB 2 and PSB 3 have two. They are not identical and not interchangeable — each PSB is a separately replaceable module with its own distinct part number. Ordering the wrong one is a wasted trip.

Supporting cast

• Riser card PART — a small board that lets an expansion card mount sideways. Why needed: expansion cards are designed to stand upright, but this chassis is flat. A riser turns the connection 90 degrees so tall cards can lie down.
• OCP (Open Compute Project) STANDARD — another rulebook, this one defining a standardized network-card slot format so cards from different vendors fit the same socket.
• DPU (Data Processing Unit) PART — a network card with its own processor. Why needed: handling network traffic consumes CPU time. A DPU does that work itself, freeing the CPU to keep feeding GPUs.
• NVMe STANDARD — the rulebook for talking to flash storage drives over PCIe. Why needed: older storage rulebooks were designed around spinning mechanical disks and were far too slow to keep flash memory busy.
• BMC (Baseboard Management Controller) PART + SOFTWARE — a small, independent computer built into the server, with its own processor and its own network port. Why needed: if the main server crashes or is powered off, you still need to see what's wrong and restart it — possibly from a thousand miles away. The BMC stays alive independently, monitors temperatures and errors, logs events, and lets you power-cycle the machine remotely. Every server manufacturer brands its own BMC implementation under a different product name; the underlying job is always the same. You'll hear about it constantly.

Part 4: Storage cabling — the confusing part, made simple

Why cabling exists here at all

In a desktop, the drive plugs into a socket on the motherboard and you're done. Why does this machine need a documented cabling scheme?

Because of the three-floor layout. The drives are at the front. The PSBs are in the basement (Layer 3). The system board is on the middle floor (Layer 2). These are physically separated by inches of metal, and there's no way to make them touch. So the connections have to be made with cables threaded between floors — and with dozens of cables and four near-identical PSBs, they need systematic names or nobody could ever service the machine.

What a backplane is

Your storage drives slide into the front of the server. Behind them sits a board covered in sockets that the drives plug into. That board is the backplane PART.

Why needed: you want to remove and replace drives from the front without opening the machine. So the sockets must be permanently mounted, facing forward, wired to the rest of the system from behind. That's all a backplane is — a fixed socket board. Drives in front, cables out the back.

Three possible configurations:

1. No backplane — no drives at all (storage lives elsewhere on the network)
2. 8x 2.5-inch — eight standard drives
3. 16x E3.S — sixteen E3.S drives (E3.S STANDARD is a newer physical form-factor specification designed specifically for flash storage — slimmer, denser, better airflow than the older 2.5-inch shape)

The connector naming convention — crack this once

Connector names on machines like this look like alphabet soup: CTRL_SRC_PA1, CTRL_DST_PA2, BP_DST_PA1. They're actually a strict system:

• SRC = source — where the cable starts
• DST = destination — where the cable arrives
• CTRL = controller — the PSB end
• BP = backplane
• PA / PB = which port group
• 1 / 2 = which connector within that group

So CTRL_SRC_PA1 reads as: on the controller/PSB, the source connector, port group A, number 1.

Why bother? Because there are four visually identical PSBs and dozens of near-identical connectors. Without a naming system, a repair would be guesswork, and a single swapped cable produces a machine that looks fine and doesn't work. (Exact label spellings vary by manufacturer — check your own service manual for the specific strings, but this source/destination/group logic is what you're looking at.)

Every PSB has two separate cabling jobs

Keep these completely separate in your head. They serve different purposes and behave differently.

Job 1 — connect to the storage backplane (uses CTRL_SRC_...)

Purpose: give the GPUs' host system a path to read training data off the drives.

This changes with configuration, because more drives need more bandwidth:

Simple rule: more drives, more cables.

Job 2 — connect to the system board (uses CTRL_DST_...)

Purpose: the uplink to the CPU — the switchboard's connection back to the decision-maker.

This never changes. In every configuration, every PSB always has two cables to the system board: CTRL_DST_PA1 and CTRL_DST_PA2. It never changes because the CPU always needs the same path, regardless of how many drives are installed.

Which backplane feeds which PSB

• Right backplane → PSB 1 and 2
• Left backplane → PSB 3 and 4

Where the cables land — and why this determines repair difficulty

• PSB 2 and PSB 3 → route up through the gap in front of the system board → connect to the FRONT of the system board. Reachable.
• PSB 1 and PSB 4 → route out into a side cable channel, then up through the bus bars → connect to the REAR of the system board. You must remove the system board to reach that channel.

Why the difference? Geometry. PSBs 2 and 3 sit in the middle, directly below the front edge of the system board, so there's a straight shot upward. PSBs 1 and 4 sit at the outer edges, where there's no vertical path — so their cables run sideways along the chassis wall first, then up. That side channel is buried underneath the system board.

Practical consequence: a PSB 1 or PSB 4 cable job is major surgery; a PSB 2 or 3 job isn't. Knowing which you're facing before you start changes how you plan the repair.

System board connectors are typically labeled something like SL1_CPU2_PA1 — SL for SlimLine (the connector type STANDARD), then which CPU, then which port group.

Getting physical access

1. Slide the drive cage forward
2. Loosen the thumbscrews on both sides of the PSB cover and lift it off
3. The four PSB modules are now exposed underneath

The optional front RAID controller

Why it might exist: by default each drive is separate, and if one dies its data is gone. Many customers want drives combined so that data survives a failure, or so several drives work together for speed. That combining technique is called RAID (Redundant Array of Independent Disks) STANDARD/technique — note it's a method, not an object.

Doing RAID takes computation. You can make the CPU do it (costing CPU time) or add a dedicated chip. That dedicated chip is a RAID controller card PART. Front-mounted variants are available on the 8x 2.5-inch configuration only. It bolts to the PSB cover and sits below the drive cage — so slide the cage forward to see it.

Its connectors: BP_DST_SA1, BP_DST_SA2, BP_PWR_CTRL, a battery cable, and CTRL_DST_PA1 / CTRL_DST_PB1 going to the system board.

Why a battery cable? RAID controllers hold data briefly in their own memory before writing it to disk. If power is lost in that instant, that data vanishes and the drive contents become inconsistent. The battery keeps that memory alive until power returns and the write completes.

Part 5: Cooling — why temperature is performance

Why this section matters more than it looks

Electricity passing through a chip becomes heat — not as a defect, but as unavoidable physics. Eight GPUs at 700W produce heat like a bank of space heaters.

And chips have a built-in self-protection reflex: past a certain temperature, they throttle — deliberately slow themselves down to generate less heat and avoid permanent damage.

Follow that through: a hot GPU is a slow GPU. Cooling isn't a maintenance chore separate from performance. In this machine, cooling is performance. They're the same subject.

The two numbers to know

Both apply to the 500W A100 and the 700W H100/H200/H800 configurations alike.

"Inlet temperature" MEASURE means the temperature of air entering the front of the server. Air flows front → back, picking up heat as it travels. So the inlet number is your starting point — if the room air is already warm, you've lost before the air even reaches a GPU. This is exactly why "check the ambient inlet temperature first" is the opening move in GPU troubleshooting.

How heat actually leaves the box

• 6 system board fans PART (60×60×56mm), mid-tray, cooling CPUs and memory
• 10 GPU fans PART (80×80×105mm), rear
• All hot-swappable — replaceable while the server runs, because you can't shut down a production machine to change a fan
• 3 air shrouds PART (system board, OCP, DPU) — shaped plastic baffles

Why shrouds exist: air, like water, takes the path of least resistance. Given an open gap, it will flow straight through the empty space and out the back, touching nothing hot. A shroud blocks the easy paths so air is forced over the components that need cooling.

Critical consequence: if no OCP card is installed, you must fit the OCP blank. An empty slot is an easy path — air escapes through it instead of cooling anything downstream. A sealed airflow path is the entire mechanism; one open hole defeats it.

Heat sinks and why the removal procedure is fussy

A heat sink PART is a finned metal block clamped onto a chip. Why needed: the chip's own surface is small, so it can't shed heat fast enough. The heat sink pulls heat out and spreads it across a large finned area for the fans to blow across.

Removing one is delicate:

• CPU heat sinks: 2 × 2U HPR type, up to 350W. There are anti-tilt wires — lock them before loosening the screws, unlock them when lifting. Why: a CPU sits on hundreds of tiny spring pins. If the assembly lifts at an angle, it bends them, and a bent-pin socket means a new system board. The wires keep the lift straight.
• GPU heat sinks: 4 front + 4 rear, 4U HPR, up to 500W. Order matters: three rotations each on screws 1-2-3-4, then back to screw 1 and fully loosen. Why: fully undoing one screw first tilts the block and puts uneven stress on the chip. Backing them off evenly keeps pressure balanced.
• NVLink heat sinks: 6 of them, up to 70W, final torque 6 ±4% lbf-in (0.68 N·m). Why torque is specified: too loose and there's an air gap that ruins heat transfer; too tight and you crack the chip. (NVLink STANDARD is NVIDIA's own rulebook for GPU-to-GPU communication — it lets GPUs exchange data directly with each other instead of routing everything through the CPU.)
• On newer GPU generations, GPU and heat sink ship as one module — you never separate them, which removes most of this risk.

Liquid cooling

This class of server is commonly air-cooled only in its base configuration — no liquid cooling option. But other machines in the same broader product family use liquid, and two rules carry over:

• Near leak sensor cables, wear vinyl gloves. Why: those sensors detect the presence of moisture. The natural moisture on your fingers registers as moisture. You'd trigger a false leak alarm or damage the sensor.
• At boot the system may ask you to select a cooling mode. Choose "No" on an air-cooled unit, since it has no liquid cooling module. Why it matters: choosing "Yes" configures the system to expect liquid-cooling temperature ranges, and it then floods the System Event Log with inlet temperature errors that aren't real.

Part 6: Checking the machine's health from Linux

Everything here is typed at a terminal as root — the administrator account with unrestricted permissions. Why root: these commands read low-level hardware information that ordinary user accounts are blocked from seeing.

The command pattern used throughout

• lspci SOFTWARE = "list PCI." It asks the system to enumerate every device attached to the internal PCIe bus and prints them. The raw output is long and messy.
• grep SOFTWARE = a text filter. It reads lines of text and prints only those matching a pattern.
• | = a pipe. It takes the output of the command on the left and feeds it as input to the command on the right.

So lspci | grep -i "nvidia" means: list every device, then show me only the lines containing "nvidia." (-i = ignore capitalization.)

That one pattern — generate everything, then filter — is the basis of nearly every diagnostic command below.

Check 1 — Are all eight GPUs present?

Why first: it's the cheapest possible check and it separates "the GPU is broken" from "the system can't see the GPU at all," which are very different problems.

Those leading codes (01:00.0, 41:00.0) are bus addresses MEASURE — the machine's internal street addresses for each device.

Count the lines. If you expect eight and count seven, you've located the problem in one command.

Check 2 — Are they running at full speed?

Why this check exists: a GPU can be present and appear fine while connected at reduced speed or on half its lanes — from a poorly seated connector, a damaged cable, or a link that failed to negotiate properly at boot. The machine still works. It's just quietly much slower, and nothing obviously alerts you.

Piece by piece: -vvv = very verbose, print all detail. -A100 = also show the 100 lines after each match (the link details sit below the device name). Then filter for LnkSta (link status) and slot number.

Healthy output:

16GT/s = full Gen4 speed. x16 = all sixteen lanes. Anything less — 8GT/s, or x8 — means a degraded link. Reseat and investigate the physical connection.

Check 3 — Network card drivers

First, the distinction people mix up constantly:

• Firmware SOFTWARE — software stored inside the hardware device itself, on a small memory chip on the card. It runs on the device's own processor and controls how the device behaves internally. It's there even with no operating system installed.
• Driver SOFTWARE — software that runs on the main CPU, as part of the operating system. It's the translator that lets the OS issue commands the device understands.

Why the update order matters: new drivers are written and tested against current firmware. Install a new driver on old firmware and you get mismatched expectations — which is why the rule is always firmware first, driver second.

Linux names network ports things like eno3, eno4, eno1np0, ens3f0. Why the odd names: they're generated from the port's physical location on the machine, so the same physical port always gets the same name even if cards are added or removed. Predictability beats readability.

ethtool -i prints the driver name (e.g. tg3), its version, and the firmware version.

Full update workflow:

1. Update the server firmware first — always, for the reason above
2. lspci | grep Ethernet — locate the cards
3. ip link — identify device names
4. ethtool -i eno4 — record the current version
5. …install the new driver…
6. reboot — why: a driver is loaded into the operating system at startup; the old one stays resident in memory until a restart clears it
7. ethtool -i eno4 again — confirm the version actually changed (example: 3.137 → 3.139b)

Step 7 is the one people skip. Never skip it. A silently failed install looks identical to a successful one until you check the version number.

Part 7: When a GPU fails — the six-step procedure

The rule that comes before everything else

On this class of server: do not replace hardware, do not dispatch labor, and do not have the customer troubleshoot internally — without a field engineer engaged.

A field engineer ROLE is a hardware-vendor specialist certified on this equipment.

Why such a strict rule? Two reasons. The parts cost enormous sums, so a wrong guess is expensive. And the components are deeply interdependent — a symptom that looks like a dead GPU is frequently a failed CPU, a mis-flashed switch firmware, or a hot room. Swapping the obvious part often fixes nothing and destroys the evidence needed to find the real fault.

This is the most likely thing you'll be tested on and the most important thing in practice.

The shape of the procedure

Notice the logic: gather evidence → verify configuration → update software → try the simple physical fix → run deep diagnostics → escalate. Cheapest, least invasive, and least destructive first. Every step preserves your ability to do the next one.

Step 1 — Gather Reports

Collect a debug support-report bundle — logs and configuration data. If it looks GPU-related, also run:

• nvidia-bug-report.sh SOFTWARE — NVIDIA's log collector
• sosreport SOFTWARE — the Linux system log collector

Do this while the server is still in the faulted state. Why: rebooting clears the volatile logs and resets the hardware state. The evidence exists only while the fault is live. Resist the instinct to restart first — that instinct destroys the only information that can identify the cause.

Step 2 — Verification

Confirm the system configuration is correct — above all, that the correct PCIe switch firmware is applied.

Why this rates its own step: the PCIe switch is the traffic controller between the CPU and every GPU. Wrong or outdated switch firmware produces symptoms — missing GPUs, degraded links, random errors — that are indistinguishable from dead hardware. Verifying it here prevents replacing a perfectly good GPU.

Step 3 — Update Firmware

Bring firmware current, noting these quirks:

• The PSBs are not reported in the BMC or in support-report bundles. Why it matters: they're invisible to your normal tooling, so nothing will remind you they exist. You must update them deliberately. This catches people out constantly.
• CPLD updates must be applied by themselves. A CPLD PART is a small programmable logic chip handling low-level board control; updating one alters fundamental board behavior mid-process, so it's isolated from other updates.
• Firmware bundles are only executable from within the OS.
• An update can take ten minutes or more and needs a power cycle to take effect.
• Separate firmware packages exist per hardware configuration (backplane type, chassis variant) — always match your exact configuration before applying one.

Step 4 — Power Cycle

Not a reboot. A real power removal:

• Power down and physically remove the AC cords for more than ten minutes
• If you can't physically pull them, run a full AC power cycle from the BIOS setup menus

Why ten minutes, and why pull the cords? Even "off," a server keeps parts of itself powered — the BMC stays alive, capacitors hold charge, and chips retain internal state. A reboot doesn't clear any of that. Removing AC entirely and waiting lets residual charge drain so components genuinely reinitialize from scratch rather than resuming from a stuck state. It sounds superstitious. It isn't.

Step 5 — dcgmi diag

DCGM (Data Center GPU Manager) SOFTWARE is NVIDIA's health-testing tool. Where lspci only asks "are you there and how are you connected," DCGM actively exercises the GPUs — running real calculations and checking the answers — and reports pass/fail.

Start with -r 1. Escalate to -r 4 when you need the complete picture and can spare the time.

Step 6 — Additional Diagnostics

The field engineer may pull in deeper factory-level diagnostics that go beyond what's exposed in the field, and beyond what this guide covers.

Troubleshooting habits that save trips

• Check ambient inlet temperature first for any GPU issue on this class of hardware. Cheapest possible check, surprisingly common root cause — and it explains "the GPU is slow" without any hardware being broken at all.
• If the BMC reports a failed GPU, reboot the BMC and see whether the entry persists. Why: the BMC is itself a small computer that can glitch. Sometimes the report is wrong, not the GPU.
• Each GPU has an affinity to one of the CPUs — it's reached through a specific CPU. If that CPU is dead, every GPU behind it disappears from the system, and the server reports a missing GPU error. So verify both CPUs are functional before condemning a GPU. This misdirection is a classic: the error names the wrong component.
• For software issues, check whether the NVIDIA driver is compatible with both the OS and the hardware, and whether the management/software engine versions are current and compatible.

Part 8: Cheat sheet

Numbers

Commands

Glossary — with what kind of thing each one is

The five things most likely to be tested

1. Engage a field engineer before any hardware action on this class of server.
2. Graceful power-capping under PSU loss is typically NVIDIA-only — AMD and Intel variants often don't have it.
3. PSBs aren't reported in the BMC or support bundles — update their firmware deliberately.
4. PSB 1 and 4 → rear of the system board (system board removal required); PSB 2 and 3 → front.
5. 35°C is safe; 38°C is the warning threshold.

Where to go next

Natural follow-ons: PSU and slot population rules (which cards go in which slots and in what priority order — driven by cooling), the teardown cable part-number mapping (matching every cable off the PDB to its replacement part), and the AMD GPU variant of this server class (OAM modules instead of the NVIDIA baseboard, GPU numbering starting at 0 instead of 1, rocm-smi instead of nvidia-smi, and AMD's own field health-check tooling).

Each of those is detail layered onto the structure you now have. The structure was the hard part.

Architecture and procedures generalized from field experience with 8-GPU HGX-class servers across multiple manufacturers; always confirm exact model numbers, connector labels, and firmware tool names against your own hardware's official documentation. See also: NVIDIA XID error codes explained. Independent training resource — not affiliated with, sponsored by, or endorsed by NVIDIA Corporation or any server manufacturer.