highPerformance

Training job hangs

A training run stops advancing steps with no crash — throughput goes to zero.

Scope
Could be collectives, storage (checkpoint), or a deadlocked rank.
Check recent changes
  • ·Scale-up
  • ·New checkpoint path
  • ·Fabric changes
Diagnostics

Find the stuck rank

text
# Compare per-rank step counters; the lagging rank is the suspect.
# py-spy dump --pid <trainer pid> on that rank shows where it's blocked.
Likely causes & fixes
  • high likelihoodCollective hang (see NCCL/RCCL workflow)

    Fix · Follow the NCCL/RCCL errors workflow.

  • medium likelihoodCheckpoint I/O deadlock/stall

    Fix · Check storage throughput; reduce checkpoint frequency.

  • low likelihoodData-loader deadlock

    Fix · Inspect worker processes; adjust worker count.

Validation
  • ✓Steps advance again at expected throughput
Escalation

Repeated hangs isolated to one node → hardware/fabric review.

Data to collect first
  • ·Per-rank step/throughput
  • ·Stack dumps of stuck ranks
  • ·Checkpoint I/O metrics
Related alerts
training-stalled