highDetection & drivers

Container can't access the GPU

The pod schedules onto a GPU node but the process can't see or use the device.

Scope
Node sees the GPU and Kubernetes scheduled it, but the container runtime isn't injecting it.
Check recent changes
  • ·Runtime/toolkit changes
  • ·Custom image without CUDA/ROCm
  • ·RuntimeClass changes
Diagnostics

Can the container see the GPU?

bash
kubectl exec <pod> -- nvidia-smi

Expect: GPUs listed. 'command not found' or no devices means the runtime didn't inject them.

Did the pod actually request a GPU?

bash
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources.limits}'

Expect: limits include nvidia.com/gpu or amd.com/gpu.

Likely causes & fixes
  • high likelihoodPod didn't request a GPU resource

    Fix · Add nvidia.com/gpu / amd.com/gpu to resources.limits.

  • high likelihoodNVIDIA container runtime not the default

    Fix · Ensure the operator/toolkit set the runtime; on RKE2 point CONTAINERD_CONFIG at the RKE2 path.

  • medium likelihoodImage lacks CUDA/ROCm userspace

    Fix · Use a CUDA/ROCm base image or mount the toolkit.

Validation
  • ✓`nvidia-smi`/`rocm-smi` works inside the container
Escalation

Runtime injects devices on some nodes but not others → inspect per-node runtime config.

Data to collect first
  • ·`nvidia-smi` inside the container
  • ·Container runtime config
  • ·Requested resources in the pod spec