Mixing GPU types and vendors in one cluster
Kubernetes happily runs heterogeneous GPU node pools — different types, and even NVIDIA + AMD together. Steer workloads with labels, taints, and affinity.
A cluster can absolutely mix GPU types (H100 for training, L40S for inference, L4 for video) and even vendors. Each node pool is labelled by its GPU (Node Feature Discovery adds labels like nvidia.com/gpu.product); you taint pools by GPU type and target workloads with a matching nodeSelector/affinity plus tolerations, so each job lands on the right hardware.
Mixing vendors works too: run both the NVIDIA GPU Operator and the AMD GPU Operator in the same cluster — each manages only its own nodes and advertises its own resource (nvidia.com/gpu vs amd.com/gpu). A pod requests whichever it needs.
The one hard rule: don't mix GPU types within a single distributed-training job. Collective operations (all-reduce) assume matched GPUs and interconnects; a slow or differently-sized GPU stalls the whole ring. Keep each training job on one homogeneous pool, and mix across pools/workloads instead. The Heterogeneous Cluster Designer models exactly this.