Live Telemetry
Current fleet state from the configured telemetry provider — per-cluster utilization, thermals, power, and ECC, with alerts assessed against the reference bands and deep-linked to their troubleshooting workflows.
Provider sample — a synthetic snapshot derived from the sample fleet + GPU catalog (no cluster required). Set TELEMETRY_PROVIDER=prometheus to scrape a real DCGM/ROCm Prometheus.
1,048
9 clusters
65%
fleet-wide
379 kW
board power
1
1 clusters
108
2 clusters
1 / 0
uncorrectable / capped
Uncorrectable ECC error
criticalprod-train-us · prod-train-us-node-06 · gpu21 uncorrectable ECC error(s) — data-integrity risk; drain and diagnose the GPU.
GPU memory pressure
warningprod-train-us · prod-train-us-node-01 · gpu390% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu492% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu691% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-02 · gpu791% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu391% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-03 · gpu490% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-04 · gpu394% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-05 · gpu392% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-06 · gpu090% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-06 · gpu290% framebuffer used.
GPU memory pressure
warningprod-train-us · prod-train-us-node-06 · gpu392% framebuffer used.
+ 97 more firing…
| Cluster | Vendor | Utilization | Peak temp | Power | Faults | State |
|---|---|---|---|---|---|---|
prod-train-us H100 SXM5 · 256 GPU | NVIDIA | 83% | 79°C | 155.6 kW | ECC 1 | critical |
prod-train-eu H100 SXM5 · 128 GPU | NVIDIA | 77% | 77°C | 74.3 kW | — | watch |
prod-train-us2 Instinct MI300X · 64 GPU | AMD | 67% | 79°C | 36.1 kW | — | watch |
prod-inf-us L40S · 192 GPU | NVIDIA | 64% | 71°C | 49.4 kW | — | healthy |
prod-inf-eu L40S · 96 GPU | NVIDIA | 58% | 68°C | 23.2 kW | — | healthy |
rag-us A100 80GB SXM · 64 GPU | NVIDIA | 71% | 75°C | 20.1 kW | — | healthy |
vision-us L4 · 120 GPU | NVIDIA | 54% | 67°C | 5.7 kW | — | healthy |
dev-us A30 · 48 GPU | NVIDIA | 22% | 52°C | 3.5 kW | — | healthy |
legacy-us A100 40GB PCIe · 80 GPU | NVIDIA | 38% | 58°C | 11 kW | — | healthy |
Snapshot 2026-09-12 08:42:32 UTC · bands from the observability reference.