highPerformance

GPU out-of-memory (OOM)

The workload is killed with a CUDA/HIP out-of-memory error.

Scope
Fits-in-memory problem; can be steady-state or a slow leak.
Check recent changes
  • ·Larger batch/context/model
  • ·Longer sequences
  • ·Serving multiple models on one GPU
Diagnostics

Memory trend

text
# DCGM_FI_DEV_FB_USED over time — steady climb ⇒ leak;
# spike at step start ⇒ batch/context too large.
Likely causes & fixes
  • high likelihoodBatch or context too large for the GPU

    Fix · Reduce batch/context; enable activation checkpointing.

  • medium likelihoodMemory fragmentation

    Fix · Tune the allocator (e.g. expandable segments).

  • medium likelihoodLeak in a long-running server

    Fix · Restart; fix the leak; cap per-request memory.

  • low likelihoodGPU simply too small for the model

    Fix · Use a larger-memory GPU or shard across GPUs.

Validation
  • ✓Workload runs to completion without OOM at steady memory
Escalation

Model can't fit even minimally → revisit GPU choice / sharding in the Recommender.

Data to collect first
  • ·Framebuffer used trend
  • ·Batch/context config
  • ·Number of models per GPU
Related alerts
high-gpu-memory