Bare metalLLM trainingHPC

LLM Training Cluster

A dense, bare-metal cluster tuned for synchronous data- and model-parallel training. The design is dominated by the east-west collective-communication fabric — everything else serves keeping the GPUs fed and the all-reduce fast.

Customize in Designer
Business objective

Train or continue-pretrain foundation models on-prem with predictable throughput and cost, without paying cloud premiums for long-running reserved capacity.

Personas
Infrastructure architectML platform engineerSREHPC admin
Key assumptions
  • ·Jobs are large and synchronous — a stalled node stalls the run.
  • ·Non-blocking or rail-optimized fabric with RDMA (InfiniBand or RoCEv2).
  • ·Checkpoints are frequent and large; storage must sustain burst writes.
  • ·Operators can plan facility power and (likely) liquid cooling.
Logical architecture

Rendering diagram…

Design

Compute

8-GPU SXM/OAM nodes with NVLink/NVSwitch or Infinity Fabric intra-node; topology-aware scheduling so ranks land on the right rails. High per-node power — plan for 700-1000W-class parts and liquid cooling.

Network

Dedicated rail-optimized fabric for training traffic (InfiniBand or lossless RoCEv2 with PFC/ECN). Separate management and storage networks. GPUDirect RDMA end-to-end; NCCL/RCCL tuned for the topology.

Storage

Parallel file system (Lustre / IBM Storage Scale / BeeGFS) sized for burst checkpoint writes; object tier for cold datasets. GPUDirect Storage where supported.

Kubernetes

Bare-metal Kubernetes with gang scheduling (Kueue or Volcano), NUMA + topology manager aligned, huge pages, and the GPU operator managing drivers/firmware.

Security

Multi-tenant queue isolation, signed images, node hardening (CIS), and secrets via Vault/ESO. Training data access is audited.

Observability

DCGM/ROCm-SMI per GPU, NCCL/RCCL profiling, fabric counters (PFC pauses, ECN marks), and job-level throughput SLOs to catch stragglers.

High availability
HA is at the job layer, not the node layer: frequent checkpointing plus elastic/resumable training so a failed node reschedules and resumes rather than losing the run.
Disaster recovery
Checkpoints and final weights replicated to a second site / object store. Dataset lineage tracked so a run is reproducible after a site loss.
Capacity
Sized by target tokens/sec and model size: GPU count from memory + compute, then network sized so all-reduce never becomes the bottleneck. Reserve headroom for failed-node reschedule.
Cost profile
CapEx-dominated. The expensive mistake is under-provisioning the fabric — GPUs idling on communication waste the most costly resource in the building.
Risks
  • highUndersized or lossy fabric turns into NCCL timeouts and collapsed throughput.
  • highPower/cooling not planned for 700-1000W parts blocks go-live.
  • mediumCheckpoint storage can't sustain burst writes, stalling training.
Implementation phases
  1. 1

    Facility + fabric

    Power, cooling, and the rail-optimized network validated with RDMA/NCCL micro-benchmarks before any workload.

  2. 2

    Platform

    Bare-metal Kubernetes, GPU operator, gang scheduler, parallel FS.

  3. 3

    Validation

    Standard training benchmark at target scale; confirm linear-ish scaling and checkpoint throughput.

  4. 4

    Production

    Onboard teams behind queues; wire straggler and fabric alerts.

Validation checklist
  • NCCL/RCCL all-reduce bandwidth within expected % of line rate at full scale.
  • No PFC pause storms or ECN saturation under sustained load.
  • Checkpoint write completes inside the checkpoint interval.
  • Failed-node job reschedule resumes from last checkpoint.
Recommended GPUs for this pattern
NVIDIA

H100 SXM5

80 GB · 700 W

NVIDIA

B200

192 GB · 1000 W

AMD

Instinct MI300X

192 GB · 750 W

AMD

Instinct MI325X

256 GB · 1000 W