mediumHardware & health

Thermal throttling

Clocks drop under load because the GPU is too hot.

Scope
Cooling/facility problem manifesting as lost throughput.
Check recent changes
  • ·Higher ambient temp
  • ·Denser packing
  • ·Cooling maintenance
Diagnostics

Confirm the throttle

bash
nvidia-smi -q -d PERFORMANCE,TEMPERATURE | grep -Ei 'throttle|temp'

Expect: 'SW Thermal Slowdown: Active' with high temperature.

Likely causes & fixes
  • high likelihoodInadequate cooling for the power density

    Fix · Improve airflow / move to liquid cooling for 700W+ parts.

  • medium likelihoodFailed fan or pump

    Fix · Replace the cooling component.

  • low likelihoodHigh ambient temperature

    Fix · Engage facility cooling.

Validation
  • ✓Temperature under the throttle threshold; clocks hold at load
Escalation

Facility-wide cooling shortfall → data-center operations.

Data to collect first
  • ·Temperature vs. clock
  • ·Per-rack thermals
  • ·Fan/pump status
Related alerts
thermal-throttle