Live Telemetry
Current fleet state from the configured telemetry provider — per-cluster utilization, thermals, power, and ECC, with alerts assessed against the reference bands and deep-linked to their troubleshooting workflows.
Provider sample — a synthetic snapshot derived from the sample fleet + GPU catalog (no cluster required). Set TELEMETRY_PROVIDER=prometheus to scrape a real DCGM/ROCm Prometheus.
1,048
9 clusters
65%
fleet-wide
378 kW
board power
1
1 clusters
103
3 clusters
0 / 2
uncorrectable / capped
GPU over thermal limit
criticalprod-train-us · prod-train-us-node-20 · gpu192°C ≥ 90°C — thermal throttling; check cooling and airflow.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu491% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu490% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu391% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu794% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu391% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-06 · gpu291% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-06 · gpu794% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-07 · gpu091% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-07 · gpu393% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-08 · gpu192% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-09 · gpu290% framebuffer used.
+ 92 more firing…
| Cluster | Vendor | Utilization | Peak temp | Power | Faults | State |
|---|---|---|---|---|---|---|
prod-train-us H100 SXM5 · 256 GPU | NVIDIA | 82% | 92°C | 154.8 kW | throttle 2 | critical |
prod-train-eu H100 SXM5 · 128 GPU | NVIDIA | 77% | 76°C | 74.2 kW | — | watch |
prod-train-us2 Instinct MI300X · 64 GPU | AMD | 68% | 79°C | 36.3 kW | — | watch |
prod-inf-us L40S · 192 GPU | NVIDIA | 64% | 71°C | 49.5 kW | — | healthy |
prod-inf-eu L40S · 96 GPU | NVIDIA | 57% | 67°C | 23.1 kW | — | healthy |
rag-us A100 80GB SXM · 64 GPU | NVIDIA | 71% | 72°C | 20 kW | — | healthy |
vision-us L4 · 120 GPU | NVIDIA | 55% | 66°C | 5.8 kW | — | healthy |
dev-us A30 · 48 GPU | NVIDIA | 25% | 50°C | 3.6 kW | — | watch |
legacy-us A100 40GB PCIe · 80 GPU | NVIDIA | 37% | 58°C | 10.9 kW | — | healthy |
Snapshot 2026-10-27 09:42:23 UTC · bands from the observability reference.