Aegis-GPU

NVIDIA Xid errors explained

What the Xid codes in dmesg mean and what NVIDIA says to do about them. Facts (mnemonics, affected GPUs, immediate and investigatory actions) are reproduced from NVIDIA's Xid Catalog, retrieved 2026-10-08; the plain-language notes are Aegis's.

Facts reproduced from NVIDIA's public Xid Catalog. Always confirm against the current NVIDIA documentation for your driver version before acting on production hardware. Scope: Ampere and newer GPUs (A100, H100, B100, GB200), including PCIe form factors. Volta and older GPUs use a separate legacy catalog.

Application faults

These usually follow a job, not the GPU: the same code on several GPUs, always with one program, points at the code. Check the application before suspecting hardware.

Xid 13: Graphics Engine Exception

ROBUST_CHANNEL_GR_EXCEPTION / ROBUST_CHANNEL_GR_ERROR_SW_NOTIFY · applies to A100, H100, B100, GB200

Raised when work on the graphics/compute engine hits an exception, commonly an illegal instruction or out-of-range access by the running program.

NVIDIA immediate action: RESTART_APP · Investigatory: WORKFLOW_XID_13

Xid 31: GPU memory page fault

ROBUST_CHANNEL_FIFO_ERROR_MMU_ERR_FLT · applies to A100, H100, B100, GB200

The GPU MMU caught an access to an invalid address: the GPU-side equivalent of a segfault, usually an application bug (CUDA "illegal memory access").

NVIDIA immediate action: RESTART_APP · Investigatory: WORKFLOW_XID_31

Xid 43: GPU stopped processing

ROBUST_CHANNEL_RESETCHANNEL_VERIF_ERROR · applies to A100, H100, B100, GB200

The driver stopped a channel (one application's work) after a fault; other work on the GPU can continue.

NVIDIA immediate action: IGNORE · Investigatory: CONTACT_SUPPORT

Xid 45: Preemptive cleanup, due to previous errors -- Most likely to see when running multiple cuda applications and hitting a DBE

ROBUST_CHANNEL_PREEMPTIVE_REMOVAL · applies to A100, H100, B100, GB200

Cleanup of an application's channels, typically after it was killed or after a preceding error. Read the codes logged just before it.

NVIDIA immediate action: WORKFLOW_XID_45 · Investigatory: Solo: RESTART_FM Not Solo: IGNORE (follow other Xid)

GPU memory (ECC and remapping)

A spectrum from "corrected and logged" to "data can no longer be trusted". Where a code sits on that spectrum decides between watching a trend, restarting one job, resetting the GPU, or contacting support.

Xid 92: High single-bit ECC error rate

EXCESSIVE_SBE_INTERRUPTS · applies to A100, H100, B100, GB200

An elevated rate of corrected single-bit errors. Nothing was corrupted, but the trend is a leading indicator worth watching.

NVIDIA immediate action: IGNORE · Investigatory: CONTACT_SUPPORT

Xid 94: Contained memory error

ROBUST_CHANNEL_CONTAINED_ERROR · applies to A100, H100, B100, GB200

An uncorrectable error was contained: only the work that consumed the bad data is affected.

NVIDIA immediate action: RESTART_APP · Investigatory: IGNORE (sympathetic)

Xid 95: Uncontained memory error

ROBUST_CHANNEL_UNCONTAINED_ERROR · applies to A100, H100, B100, GB200

An uncorrectable error could not be contained to one process, so the whole GPU state is suspect.

NVIDIA immediate action: RESET_GPU · Investigatory: IGNORE (sympathetic)

Xid 48: Double Bit ECC Error

ROBUST_CHANNEL_GPU_ECC_DBE · applies to A100, H100, B100, GB200

A double-bit (uncorrectable) ECC error. The catalog notes a GPU reset or node reboot is needed to clear it.

NVIDIA immediate action: WORKFLOW_XID_48 · Investigatory: WORKFLOW_XID_48

Xid 63: GPU memory remapping event

INFOROM_DRAM_RETIREMENT_EVENT · applies to A100, H100, B100, GB200

A memory row was marked for remapping. Per NVIDIA's memory error management docs, the remap takes effect after the next GPU reset.

NVIDIA immediate action: IGNORE · Investigatory: IGNORE

Xid 64: GPU memory remapping failure

INFOROM_DRAM_RETIREMENT_FAILURE · applies to A100, H100, B100, GB200

The remapping mechanism itself failed to record the repair.

NVIDIA immediate action: RESET_GPU · Investigatory: CONTACT_SUPPORT

Interconnect (NVLink)

Link-layer problems between GPUs. Hopper/Ampere report them as Xid 74; Blackwell reports NVLink5 problems in the 144–150 range.

Xid 74: NVLINK Error

NVLINK_ERROR · applies to A100, H100

An NVLink error on A100/H100-class systems. NVIDIA's NVLink workflow decodes the error bits to tell link-quality problems from other causes.

NVIDIA immediate action: WORKFLOW_NVLINK_ERR · Investigatory: CONTACT_SUPPORT

Xid 149: NVLINK: NETIR Error

NVLINK_NETIR_ERROR · applies to B100, GB200

NVLink5 "NETIR" errors on B100/GB200 (for example link down or bit-error events).

NVIDIA immediate action: WORKFLOW_NVLINK5_ERR · Investigatory: WORKFLOW_NVLINK5_ERR

Bus, firmware and recovery

The GPU stops answering, or the driver tells you what recovery it needs.

Xid 79: GPU has fallen off the bus

ROBUST_CHANNEL_GPU_HAS_FALLEN_OFF_THE_BUS · applies to A100, H100, B100, GB200

The system lost contact with the GPU over PCIe ("fallen off the bus").

NVIDIA immediate action: RESTART_BM · Investigatory: CONTACT_SUPPORT

Xid 119: GSP RPC Timeout

GSP_RPC_TIMEOUT · applies to A100, H100, B100, GB200

A call from the driver to the GPU System Processor (GSP) firmware timed out; the GPU can appear hung.

NVIDIA immediate action: RESET_GPU · Investigatory: INVESTIGATE_SW

Xid 154: GPU Recovery Action Changed

GPU_RECOVERY_ACTION_CHANGED · applies to A100, H100, B100, GB200

Not a fault itself: the driver states the recovery action (for example GPU reset or node reboot) that another Xid now requires.

NVIDIA immediate action: XID_154 · Investigatory: N/A Informational only regarding another Xid

All 109 in-use Xid codes

XidMnemonicDescriptionGPUsImmediate actionInvestigatory action
8ROBUST_CHANNEL_FIFO_ERROR_IDLE_TIMEOUTGPU stopped processingA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
11ROBUST_CHANNEL_GR_ERROR_MISSING_HWInvalid or corrupted push buffer streamA100, H100, B100, GB200RESTART_APPCHECK_APP/CUDA
13ROBUST_CHANNEL_GR_EXCEPTION / ROBUST_CHANNEL_GR_ERROR_SW_NOTIFYGraphics Engine ExceptionA100, H100, B100, GB200RESTART_APPWORKFLOW_XID_13
25ROBUST_CHANNEL_GR_ILLEGAL_NOTIFYInvalid or illegal push buffer streamA100, H100, B100, GB200RESTART_APPCHECK_APP/CUDA
31ROBUST_CHANNEL_FIFO_ERROR_MMU_ERR_FLTGPU memory page faultA100, H100, B100, GB200RESTART_APPWORKFLOW_XID_31
32ROBUST_CHANNEL_PBDMA_ERRORInvalid or corrupted push buffer streamA100, H100, B100, GB200RESTART_APPCHECK_APP/CUDA
37ROBUST_CHANNEL_FECS_ERR_UNIMP_FIRMWARE_METHODDriver firmware errorA100, H100, B100, GB200IGNORECHECK_APP/CUDA
38ROBUST_CHANNEL_FECS_ERR_WATCHDOG_TIMEOUTDriver firmware errorA100, H100, B100, GB200IGNORECONTACT_SUPPORT
39ROBUST_CHANNEL_CE0_ERRORCopy Engine ExceptionA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
40ROBUST_CHANNEL_CE1_ERRORCopy Engine ExceptionA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
41ROBUST_CHANNEL_CE2_ERRORCopy Engine ExceptionA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
43ROBUST_CHANNEL_RESETCHANNEL_VERIF_ERRORGPU stopped processingA100, H100, B100, GB200IGNORECONTACT_SUPPORT
44ROBUST_CHANNEL_GR_FAULT_DURING_CTXSWGraphics Engine fault during context switchA100, H100, B100, GB200IGNORECONTACT_SUPPORT
45ROBUST_CHANNEL_PREEMPTIVE_REMOVALPreemptive cleanup, due to previous errors -- Most likely to see when running multiple cuda applications and hitting a DBEA100, H100, B100, GB200WORKFLOW_XID_45Solo: RESTART_FM Not Solo: IGNORE (follow other Xid)
46ROBUST_CHANNEL_GPU_TIMEOUT_ERRORGPU stopped processingA100, H100, B100, GB200RESET_GPUCONTACT_SUPPORT
48ROBUST_CHANNEL_GPU_ECC_DBEDouble Bit ECC ErrorA100, H100, B100, GB200WORKFLOW_XID_48WORKFLOW_XID_48
54SILENT_RUNNING_PWR_REDUCED_CLOCKINGAuxiliary power is not connected to the GPU boardA100, H100, B100CHECK_MECHANICALSCONTACT_SUPPORT
60ROBUST_CHANNEL_SEC2_ERRORVideo processor exceptionA100, H100, B100, GB200RESTART_APPINVESTIGATE_SW
62PMU_HALT_ERRORInternal micro-controller halt (newer drivers)A100, H100, B100, GB200RESET_GPUCONTACT_SUPPORT
63INFOROM_DRAM_RETIREMENT_EVENTGPU memory remapping eventA100, H100, B100, GB200IGNOREIGNORE
64INFOROM_DRAM_RETIREMENT_FAILUREGPU memory remapping failureA100, H100, B100, GB200RESET_GPUCONTACT_SUPPORT
66ROBUST_CHANNEL_FECS_ERR_REG_ACCESS_VIOLATIONIllegal access by driverA100, H100, B100, GB200IGNOREINVESTIGATE_SW
67ROBUST_CHANNEL_FECS_ERR_VERIF_VIOLATIONIllegal access by driverA100, H100, B100, GB200IGNORECONTACT_SUPPORT
68ROBUST_CHANNEL_NVDEC0_ERRORNVDEC0 ExceptionA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
69ROBUST_CHANNEL_GR_CLASS_ERRORGraphics Engine class errorA100, H100, B100, GB200RESTART_APPCHECK_APP/CUDA
70ROBUST_CHANNEL_CE3_ERRORCE3: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
71ROBUST_CHANNEL_CE4_ERRORCE4: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
72ROBUST_CHANNEL_CE5_ERRORCE5: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
74NVLINK_ERRORNVLINK ErrorA100, H100WORKFLOW_NVLINK_ERRCONTACT_SUPPORT
75ROBUST_CHANNEL_CE6_ERRORCE6: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
76ROBUST_CHANNEL_CE7_ERRORCE7: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
77ROBUST_CHANNEL_CE8_ERRORCE8: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
78VGPU_START_ERRORvGPU Start ErrorA100, H100, B100, GB200UPDATE_SWFWUPDATE_SWFW
79ROBUST_CHANNEL_GPU_HAS_FALLEN_OFF_THE_BUSGPU has fallen off the busA100, H100, B100, GB200RESTART_BMCONTACT_SUPPORT
80PBDMA_PUSHBUFFER_CRC_MISMATCHCorrupted data sent to GPUA100, H100RESTART_APPCHECK_APP/CUDA
82ROBUST_CHANNEL_NVJPG0_ERRORNVJPG0 ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
83ROBUST_CHANNEL_NVDEC1_ERRORNVDEC1 ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
84ROBUST_CHANNEL_NVDEC2_ERRORNVDEC2 ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
85ROBUST_CHANNEL_CE9_ERRORCE9: Unknown ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
86ROBUST_CHANNEL_OFA0_ERROROFA ExceptionA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
88ROBUST_CHANNEL_NVDEC3_ERRORNVDEC3 ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
89ROBUST_CHANNEL_NVDEC4_ERRORNVDEC4 ErrorA100, H100, B100, GB200RESTART_APPCONTACT_SUPPORT
92EXCESSIVE_SBE_INTERRUPTSHigh single-bit ECC error rateA100, H100, B100, GB200IGNORECONTACT_SUPPORT
93INFOROM_ERASE_LIMIT_EXCEEDEDNon-fatal violation of provisioned InfoROM wear limitA100IGNORECONTACT_SUPPORT
94ROBUST_CHANNEL_CONTAINED_ERRORContained memory errorA100, H100, B100, GB200RESTART_APPIGNORE (sympathetic)
95ROBUST_CHANNEL_UNCONTAINED_ERRORUncontained memory errorA100, H100, B100, GB200RESET_GPUIGNORE (sympathetic)
96ROBUST_CHANNEL_NVDEC5_ERRORNVDEC5 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
97ROBUST_CHANNEL_NVDEC6_ERRORNVDEC6 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
98ROBUST_CHANNEL_NVDEC7_ERRORNVDEC7 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
99ROBUST_CHANNEL_NVJPG1_ERRORNVJPG1 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
100ROBUST_CHANNEL_NVJPG2_ERRORNVJPG2 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
101ROBUST_CHANNEL_NVJPG3_ERRORNVJPG3 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
102ROBUST_CHANNEL_NVJPG4_ERRORNVJPG4 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
103ROBUST_CHANNEL_NVJPG5_ERRORNVJPG5 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
104ROBUST_CHANNEL_NVJPG6_ERRORNVJPG6 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
105ROBUST_CHANNEL_NVJPG7_ERRORNVJPG7 ErrorH100, B100, GB200RESTART_APPCONTACT_SUPPORT
106SMBPBI_TEST_MESSAGESMBPBI Test MessageA100, H100, B100, GB200IGNOREIGNORE
107SMBPBI_TEST_MESSAGE_SILENTSMBPBI Test Message SilentA100, H100, B100, GB200IGNOREIGNORE
109ROBUST_CHANNEL_CTXSW_TIMEOUT_ERRORContext Switch Timeout ErrorA100, H100, B100, GB200RESET_GPUCONTACT_SUPPORT
110SEC_FAULT_ERRORSecurity Fault ErrorH100, B100, GB200RESET_GPUINVESTIGATE_SW
119GSP_RPC_TIMEOUTGSP RPC TimeoutA100, H100, B100, GB200RESET_GPUINVESTIGATE_SW
120GSP_ERRORGSP ErrorA100, H100, B100, GB200RESET_GPUINVESTIGATE_SW
121C2C_ERRORC2C ErrorGB200IGNORECONTACT_SUPPORT
126ROBUST_CHANNEL_CE10_ERRORCE10: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
127ROBUST_CHANNEL_CE11_ERRORCE11: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
128ROBUST_CHANNEL_CE12_ERRORCE12: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
129ROBUST_CHANNEL_CE13_ERRORCE13: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
130ROBUST_CHANNEL_CE14_ERRORCE14: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
131ROBUST_CHANNEL_CE15_ERRORCE15: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
132ROBUST_CHANNEL_CE16_ERRORCE16: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
133ROBUST_CHANNEL_CE17_ERRORCE17: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
134ROBUST_CHANNEL_CE18_ERRORCE18: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
135ROBUST_CHANNEL_CE19_ERRORCE19: Unknown ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
136ALI_TRAINING_FAILLink Training FailedH100RESET_GPUINVESTIGATE_LINK_SI
137NVLINK_PRIV_ERRNVLink Privilege ErrorA100, H100, B100, GB200IGNOREXID_137_FLOW
139ROBUST_CHANNEL_OFA1_ERROROFA1 ErrorB100, GB200RESTART_APPCONTACT_SUPPORT
140UNRECOVERABLE_ECC_ERROR_ESCAPEECC Unrecovered ErrorA100, H100, B100, GB200RESET_GPUCONTACT_SUPPORT
141ROBUST_CHANNEL_FAST_PATH_ERRORCUDA Fast Path ErrorH100, B100, GB200IGNORECONTACT_SUPPORT
142ROBUST_CHANNEL_NVENC3_ERRORNVENC3 ErrorGB200CONTACT_SUPPORT—
143GPU_INIT_ERRORGPU Initialization ErrorH100, B100, GB200RESET_GPUCONTACT_SUPPORT
144NVLINK_SAW_ERRORNVLINK: SAW ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
145NVLINK_RLW_ERRORNVLINK: RLW ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
146NVLINK_TLW_ERRORNVLINK: TLW ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
147NVLINK_TREX_ERRORNVLINK: TREX ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
148NVLINK_NVLPW_CTRL_ERRORNVLINK: NVLPW_CTRL ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
149NVLINK_NETIR_ERRORNVLINK: NETIR ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
150NVLINK_MSE_ERRORNVLINK: MSE ErrorB100, GB200WORKFLOW_NVLINK5_ERRWORKFLOW_NVLINK5_ERR
151ROBUST_CHANNEL_KEY_ROTATION_ERRORKey rotation ErrorH100, B100, GB200RESTART_VMCONTACT_SUPPORT
152ROBUST_CHANNEL_DLA_SMMU_ERRORDLA SMMU ErrorIGNORECONTACT_SUPPORT
153ROBUST_CHANNEL_DLA_TIMEOUTDLA timeout ErrorIGNORECONTACT_SUPPORT
154GPU_RECOVERY_ACTION_CHANGEDGPU Recovery Action ChangedA100, H100, B100, GB200XID_154N/A Informational only regarding another Xid
155NVLINK_SW_DEFINED_ERRORNVLINK: SW Defined ErrorB100, GB200RESET_GPUINVESTIGATE_SW_USER
156RESOURCE_RETIREMENT_EVENTResource Retirement EventH100, B100, GB200RESET_GPUIGNORE
157RESOURCE_RETIREMENT_FAILUREResource Retirement FailureH100, B100, GB200IGNORECONTACT_SUPPORT
158GPU_FATAL_TIMEOUTGPU Fatal TimeoutA100, H100, B100, GB200RESET_GPUCONTACT_SUPPORT
159ROBUST_CHANNEL_CHI_NON_DATA_ERRORCHI Non-Data ErrorB100, GB200CHECK_UVMSYMPATHETIC_REPORT_SOLO
160CHANNEL_RETIREMENT_EVENTChannel Retirement EventB100, GB200IGNOREINVESTIGATE_SW
161CHANNEL_RETIREMENT_FAILUREChannel Retirement FailureB100, GB200IGNOREINVESTIGATE_SW
162PSHC_REENGAGEDPower Smoothing HW Circuitry capability reengagedB100, GB200——
163PSHC_DISENGAGEDPower Smoothing HW Circuitry capability disengagedB100, GB200——
164PSHC_LOW_LIFETIMEPower Smoothing HW Circuitry low lifetime reachedB100, GB200——
165PSHC_ZERO_LIFETIMEPower Smoothing HW Circuitry lifetime exhaustedB100, GB200——
166NVLINK_SECURE_CRYPTO_ERRCC traffic seen prior to link properly being configured for encrypted trafficB100, GB200——
167PCIE_FATAL_TIMEOUTPCIE_FATAL_TIMEOUTH100, B100, GB200——
168REDUCED_GPU_MEMORY_CAPACITYErrors found in WPR (write protected region)A100, H100, B100, GB200——
169SEC2_HALT_ERRORInternal micro-controller haltH100, B100, GB200——
170NVLINK_SECURE_OTHERInterrupt seen in CC modeB100, GB200——
171UNCORRECTABLE_DRAM_ERRORAdditional to Xid 48 providing more details on particulars of fault to differentiate DRAM/SRAMA100, H100, B100, GB200——
172UNCORRECTABLE_SRAM_ERRORAdditional to Xid 48 providing more details on particulars of fault to differentiate DRAM/SRAMA100, H100, B100, GB200——

What the action names mean

NVIDIA groups recovery steps into named resolution buckets. Descriptions below are NVIDIA's.

BucketMeaning
CONTACT_SUPPORTPlease contact your support organization for further investigation.
RESTART_APPThe application should be restarted RESET_GPU or RESTART_BM is not deemed necessary.
IGNORENo Action required
WORKFLOW_XID_45Solo: RESTART_FM Not Solo: IGNORE (follow guidance in other Xid)
RESET_GPURefer to https://docs.nvidia.com/deploy/gpu-debug-guidelines/index.html for GPU Reset capabilities & limitations RESTART_BM is not deemed necessary.
WORKFLOW_XID_48Data Center Recovery Action Solo: RESET_GPU w/ 63 or 64: DRAIN_AND_RESET Investagatory Action Solo: RUN_FIELDDIAG Not Solo: String in error would tell us what unit was impacted. FB: follow Xid63/64 guidance All other SRAM: check SRAM Error Threshold flag (nvidia-smi <sram_threshold_exceeded> or NSM Msg Type 0x3, Cmd Code 0x7D, bit 0. If set RUN_FIELDDIAG
CHECK_MECHANICALSCheck to ensure that device seating and all applicable connections to it are secure.
UPDATE_SWFWUpdate Firmware and Software to latest versions XID 78: basic issues will keep vGPU functionality from being able to operate; must resolve to progress 1. Guest driver version is incompatible with the host driver * In this case error string should be "Guest driver is incompatible with host driver" 2. This vGPU type is not compatible with the guest OS type/GPU type. For example, user is trying to use a compute profile on an old Maxwell GPU on windows guest. * In this case error string should be "vGPU type is not supported"
RESTART_BMRestart bare metal, system should be restarted
WORKFLOW_NVLINK5_ERRPlease see the "XID 144-150 Decode" (was "Customer Doc 144-150")tab for further guidance in evaluating these Xids. These errors need decoding of XID message as follows to determine the resolution action: Format of the XID error message: Xid (PCI:0000:BB:DF): <Xid Number> <sub component> <fatal vs nonfatal> <Crosscontain> <injected> <link> (<intrInfo> <errorStatus> <errorDebugData[0]> <errorDebugData[1]> <errorDebugData[2]> <errorDebugData[3]> <errorDebugData[4]>) From the above message, <intrInfo>, <errorStatus> must be decoded and evaluated using "XID 144-150 Decode" to derive the final resolution.
RESTART_VMVM owning the affected GPU must be restarted RESET_GPU or RESTART_BM is not deemed necessary.
XID_154Follow XID 154 reported guidance
CHECK_UVMIf UVM/vGPU is being utilized, RESET_GPU; otherwise IGNORE
CHECK_APP/CUDAIssue likely caused by an application passing bad data or utilizing incorrect methods in communications with GPU. Some errors will contain PID that can be used to identify source of the problem. If determined to be a driver issue then REPORT_ISSUE
WORKFLOW_XID_13Repeat TPC and GPC, diff SMs: RUN_DCGMEUD (possible HW issue); if pass RUN_FIELDDIAGS Repeat TPC and GPC, single SM: RUN_DCGMEUD (possible HW issue); if pass RUN_FIELDDIAGS Solo, no burst: CHECK_APP/CUDA Not Repeat TPC and GPC: CHECK_APP/CUDA Non-prod environment: CHECK_APP/CUDA If known good APP and Solo: REPORT_ISSUE
WORKFLOW_XID_31Multiple runs needed to establish pattern Repeat MMU faults to same GPU (via PCI-ID): RUN_DCGMEUD (possible HW issue); if pass RUN_FIELDDIAGS Repeat MMU faults to diff GPU (via PCI-ID): CHECK_APP/CUDA Solo, no burst: CHECK_APP/CUDA If known good APP: REPORT_ISSUE
Solo: RESTART_FM Not Solo: IGNORE (follow other Xid)Solo: RESTART_FM Not Solo: IGNORE (follow other Xid)
INVESTIGATE_SWThere is a problem with either user or NVIDIA software that needs to be investigated further. In many cases the user software may be making calls to illegal areas, poorly structured commands or other issues. This may also be a problem with NVIDIA software in which case an issue should be reported. In many cases there may be a PID that could be tracked back to the offending, originating entity.
IGNORE (sympathetic)This is a sympathetic error that is expected to be seen with other conditions. Resolution for the other errors should be undertaken first. If this error was seen independently or all other resolutions aren't suffficient then REPORT_ISSUE.
XID_137_FLOWThis event is logged when a fault is reported by the remote MMU, such as when an illegal NVLink peer-to-peer access is made by an applicable unit on the chip. Typically these are application-level bugs. When this event is logged, NVIDIA recommends the following: • Run the application in cuda-gdb or cuda-memcheck • Note: The cuda-memcheck tool instruments the running application and reports which line of code performed the illegal read. or • Run the application with CUDA_DEVICE_WAITS_ON_EXCEPTION=1 and then attach later with cuda-gdb File a bug if the previous two come back inconclusive to eliminate other possible causes.
N/A Informational only regarding another XidOther issues need to be addressed. This Xid in informational only and is always expected to be seen with another Xid requiring a recovery action.
INVESTIGATE_SW_USERInvestigate SW or user initiated if unexpected
SYMPATHETIC_REPORT_SOLOThis is a sympathetic error that is expected to be seen with other conditions. Resolution for the other errors should be undertaken first. If this error was seen independently or all other resolutions aren't suffficient then REPORT_ISSUE.
XID_154_EVALIf an XID 154 is seen along with this error, take that action. If no XID 154 present, RESTART_APP
REPORT_ISSUE (if seen >1 per day)REPORT_ISSUE (if seen >1 per day)
INVESTIGATE_PEER_DEVICEA peer device experience the issue as decoded in the Xid144-150 table (Xid 149 in particular). Based upon the note column: Received_TS1: will be seen when peer link down reason is unknown Peer_side_down_to_sleep_state: investigate peer software and users if unexpected Peer_side_down_to_disable_state: investigate peer software and users if unexpected Peer_side_down_to_disable_and_port_lock: investigate peer software and users if unexpected Peer_side_down_due_to_thermal_event: check switch cooling Peer_side_down_due_to_force_event: investigate peer software and users if unexpected Peer_side_down_due_to_reset_event: investigate peer software and users if unexpected
INVESTIGATE_HOSTCheck other logs as this is likely a secondary indicator of some action or fault (may be OOB).

Practise these: the lab simulator injects Xid 48, 74 and 79 scenarios, and NCA-AIIO practice includes Xid questions.