Live Telemetry
Current fleet state from the configured telemetry provider — per-cluster utilization, thermals, power, and ECC, with alerts assessed against the reference bands and deep-linked to their troubleshooting workflows.
Provider sample — a synthetic snapshot derived from the sample fleet + GPU catalog (no cluster required). Set TELEMETRY_PROVIDER=prometheus to scrape a real DCGM/ROCm Prometheus.
1,048
9 clusters
65%
fleet-wide
377 kW
board power
1
1 clusters
98
3 clusters
0 / 1
uncorrectable / capped
GPU over thermal limit
criticalprod-train-us · prod-train-us-node-22 · gpu592°C ≥ 90°C — thermal throttling; check cooling and airflow.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu390% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu290% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu490% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu693% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu791% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu093% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu191% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu191% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu290% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu390% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-05 · gpu791% framebuffer used.
+ 87 more firing…
| Cluster | Vendor | Utilization | Peak temp | Power | Faults | State |
|---|---|---|---|---|---|---|
prod-train-us H100 SXM5 · 256 GPU | NVIDIA | 82% | 92°C | 154.3 kW | throttle 1 | critical |
prod-train-eu H100 SXM5 · 128 GPU | NVIDIA | 77% | 76°C | 73.8 kW | — | watch |
prod-train-us2 Instinct MI300X · 64 GPU | AMD | 68% | 78°C | 36.8 kW | — | watch |
prod-inf-us L40S · 192 GPU | NVIDIA | 64% | 71°C | 49.3 kW | — | healthy |
prod-inf-eu L40S · 96 GPU | NVIDIA | 57% | 67°C | 22.9 kW | — | healthy |
rag-us A100 80GB SXM · 64 GPU | NVIDIA | 71% | 74°C | 20.1 kW | — | healthy |
vision-us L4 · 120 GPU | NVIDIA | 55% | 65°C | 5.8 kW | — | watch |
dev-us A30 · 48 GPU | NVIDIA | 24% | 52°C | 3.5 kW | — | healthy |
legacy-us A100 40GB PCIe · 80 GPU | NVIDIA | 36% | 59°C | 10.8 kW | — | healthy |
Snapshot 2026-07-26 20:59:24 UTC · bands from the observability reference.