criticalDetection & drivers

GPU not detected in Kubernetes

A GPU node reports zero allocatable nvidia.com/gpu (or amd.com/gpu), so nothing can schedule on it.

Scope
One node, a subset, or all GPU nodes — determine the blast radius first.
Check recent changes
  • ·Kernel or driver upgrade
  • ·OS patch/reboot
  • ·GPU operator upgrade
  • ·New node just joined
Guided diagnosis

On the host, does `nvidia-smi` / `rocm-smi` list the GPUs?

Diagnostics

Is the GPU allocatable to the scheduler?

bash
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'

Expect: A number > 0 per GPU node. Blank/0 means enablement is broken.

Are the operator/device-plugin pods healthy?

bash
kubectl -n gpu-operator get pods -o wide | grep <node>

Expect: device-plugin and driver pods Running on the node.

Does the host see the GPU at all?

bash
ssh <node> nvidia-smi   # or rocm-smi

Expect: GPUs listed. If this fails, it's a host/driver problem, not Kubernetes.

Likely causes & fixes
  • high likelihoodDevice-plugin/driver pod crashed or never started

    Fix · Restart the pod; check its logs for the root error.

  • high likelihoodDriver/kernel mismatch after an upgrade

    Fix · Align the driver with the running kernel (operator rebuild or pin versions).

  • medium likelihoodGPU fell off the PCI bus (hardware)

    Fix · Drain and reboot; if it persists, RMA.

  • low likelihoodNode labels/RuntimeClass missing

    Fix · Verify NFD labels and the NVIDIA/AMD runtime configuration.

Validation
  • ✓Allocatable GPU count returns to expected
  • ✓A test GPU pod schedules and runs `nvidia-smi`/`rocm-smi`
Escalation

Host doesn't see the GPU after reboot → hardware team / vendor RMA.

Data to collect first
  • ·Node allocatable resources
  • ·Device-plugin/operator pod status
  • ·Host-level GPU visibility
  • ·Kernel logs
Related alerts
gpu-unavailable