ConceptSREs
The GPU health signals that matter
ECC errors, Xid codes, thermal throttling, and utilization tell you whether a GPU is healthy, degrading, or failing.
Uncorrectable ECC errors and fatal Xid codes (79, 48, 94) mean failing hardware — drain the node immediately and RMA. Rising correctable ECC is an early warning to schedule maintenance.
Thermal and power throttling silently cut throughput; correlate temperature with clock speed. Persistently low utilization usually means a data or communication bottleneck, not spare capacity.
Export these with DCGM (NVIDIA) or ROCm-SMI (AMD); the Observability section has the metric bands and alert catalog.
observabilitydcgmhealth