mediumStorage

Storage bottleneck

GPUs stall waiting on storage — checkpoints or dataset reads can't keep up.

Scope
I/O path between compute and storage.
Check recent changes
  • ·Bigger checkpoints
  • ·More concurrent readers
  • ·Storage backend change
Diagnostics

Is I/O the wait?

text
# Low GPU util + high checkpoint/read time ⇒ storage-bound.
# Check FS throughput vs. the parallel-FS spec.
Likely causes & fixes
  • high likelihoodStorage can't sustain burst checkpoint writes

    Fix · Move to a parallel file system sized for burst.

  • medium likelihoodNetwork path to storage saturated

    Fix · Add a storage network or bring data closer.

  • low likelihoodToo-frequent checkpointing

    Fix · Lengthen the interval / async checkpoint.

Validation
  • ✓Checkpoint completes within its interval; GPU util recovers
Escalation

Parallel-FS itself underperforms spec → storage engineering.

Data to collect first
  • ·Storage read/write throughput and latency
  • ·Checkpoint duration vs. interval
  • ·Network path to storage
Related alerts
checkpoint-delay