Deploy

Cluster Bootstrap

Describe your target environment. The wizard generates a phased bootstrap plan — from hardware validation through GPU enablement to workload validation — with parameterized scripts and manifests, per-phase validation, and rollback. Nothing runs against a live cluster.

Bootstrap plan

Kubespray · H100 SXM5

Self-managed12 phases
  1. 1

    Hardware & firmware validation

    low risk

    Verify GPUs are seated, firmware/BMC is current, and power/cooling meet the node's draw.

    • ·Confirm each worker sees its H100 SXM5 GPUs.
    • ·Update BMC/BIOS and GPU firmware to a supported baseline.
    • ·Verify facility power for ~700 W/GPU and cooling (air or liquid).
    Detect GPUs on each host
    lspci | grep -Ei 'nvidia|amd/ati'
# Expect one line per installed GPU
    Validation
    • All GPUs enumerate on every worker host.

    Rollback: None — read-only validation.

  2. 2

    OS & kernel preparation

    medium risk

    Prepare the host OS for GPUs and Kubernetes: modules, IOMMU, huge pages, swap.

    • ·Install a supported Ubuntu kernel and headers.
    • ·Blacklist the nouveau driver so the GPU driver can load.
    • ·Enable IOMMU and configure huge pages for RDMA/large models.
    • ·Disable swap (required by kubelet).
    Host prep
    # Disable swap
sudo swapoff -a && sudo sed -i '/ swap / s/^/#/' /etc/fstab
# Blacklist nouveau (driver installed later by the GPU operator)
echo -e 'blacklist nouveau\noptions nouveau modeset=0' | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u || sudo dracut -f
# Enable IOMMU (GRUB) then reboot
# add: intel_iommu=on iommu=pt
    Validation
    • Node reboots cleanly; nouveau not loaded (NVIDIA); swap off.

    Rollback: Revert GRUB/modprobe changes and re-enable swap; reboot.

  3. 3

    Container runtime

    medium risk

    Install containerd and prepare it for the GPU container runtime.

    • ·Install containerd and set the systemd cgroup driver.
    • ·The GPU operator will register the GPU runtime; no manual toolkit install needed.
    containerd
    sudo apt-get install -y containerd.io   # or dnf on RHEL/SLES
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
    Validation
    • `systemctl status containerd` is active on all nodes.

    Rollback: Stop/remove containerd; revert config.

  4. 4

    Kubernetes control plane

    high risk

    Bring up a highly-available control plane (3 nodes + LB).

    • ·Stand up a load balancer VIP for the API server.
    • ·Bootstrap Kubespray across 3 control-plane nodes.
    • ·Save the join command/token for workers.
    Initialize control plane
    sudo kubeadm init --control-plane-endpoint "$LB_VIP:6443" --upload-certs \
  --pod-network-cidr=10.244.0.0/16
mkdir -p ~/.kube && sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
    Validation
    • `kubectl get nodes` shows control-plane node(s) Ready.
    • `kubectl get pods -A` core components running.

    Rollback: `kubeadm reset` (or distro teardown) on the control-plane node(s).

  5. 5

    CNI (pod networking)

    medium risk

    Install a container network so pods can communicate.

    • ·Install Cilium (or Calico).
    • ·For RDMA training, add Multus + SR-IOV CNI later.
    Install Cilium
    cilium install --version 1.16.5
cilium status --wait
    Validation
    • CNI pods Ready; a test pod gets an IP and can reach the API.

    Rollback: Uninstall the CNI (`cilium uninstall`).

  6. 6

    Join GPU workers

    high risk

    Join 4 GPU worker node(s) to the cluster.

    • ·Run the saved join command on each GPU worker.
    • ·Label and taint them so only GPU workloads schedule there.
    Label & taint GPU pools
    # After 'kubeadm join' on each worker:
kubectl label nodes -l '!node-role.kubernetes.io/control-plane' \
  gpu.vendor=nvidia node-type=gpu --overwrite
kubectl taint nodes -l node-type=gpu \
  nvidia.com/gpu=present:NoSchedule --overwrite
    Validation
    • `kubectl get nodes` shows 4 worker(s) Ready.
    • GPU nodes carry the node-type=gpu taint.

    Rollback: `kubectl drain` then `kubectl delete node`; `kubeadm reset` on the worker.

  7. 7

    GPU drivers & device plugin

    high risk

    Install the NVIDIA GPU operator to manage drivers, the container runtime, and the device plugin.

    • ·Install the NVIDIA GPU Operator via Helm.
    • ·Let the operator(s) pull upstream images.
    • ·Standard containerd — the operator configures the GPU runtime.
    NVIDIA GPU Operator
    helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace
    Validation
    • Operator pods Running
    • Nodes advertise `nvidia.com/gpu` as allocatable.

    Rollback: helm uninstall the operator(s); the driver DaemonSet(s) are removed.

  8. 8

    GPU partitioning (optional)

    medium risk

    Enable MIG on H100 SXM5 to share each GPU across smaller workloads.

    • ·Set a MIG strategy on the operator's ClusterPolicy.
    • ·Label nodes with the desired MIG profile.
    Enable MIG (single strategy)
    kubectl label nodes -l node-type=gpu \
  nvidia.com/mig.config=all-1g.10gb --overwrite
# The operator reconfigures the GPUs into MIG instances.
    Validation
    • Allocatable resources show MIG instances (e.g. nvidia.com/mig-1g.10gb).

    Rollback: Set `nvidia.com/mig.config=all-disabled` to return to full GPUs.

  9. 9

    Observability

    low risk

    Install metrics + GPU telemetry (DCGM for NVIDIA / ROCm exporter for AMD).

    • ·Install kube-prometheus-stack.
    • ·The GPU operator ships DCGM-exporter; add its ServiceMonitor.
    • ·Import GPU dashboards into Grafana.
    kube-prometheus-stack
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace
    Validation
    • GPU utilization/temperature/power metrics appear in Prometheus.
    • Grafana dashboards render for GPU nodes.

    Rollback: helm uninstall kps; remove ServiceMonitors.

  10. 10

    GitOps (Argo CD)

    medium risk

    Bootstrap declarative delivery so future changes flow through Git.

    • ·Install Argo CD.
    • ·Point it at your platform repo.
    • ·Connect to your Git provider.
    Argo CD
    kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
    Validation
    • Argo CD reconciles the platform repo with no drift.

    Rollback: Uninstall Argo CD; cluster state remains.

  11. 11

    Security policies

    medium risk

    Enforce admission policy and pod security before workloads land.

    • ·Install a policy engine (Kyverno or Gatekeeper).
    • ·Apply pod-security and signed-image policies.
    • ·Restrict registries and enforce non-root.
    Install Kyverno
    helm repo add kyverno https://kyverno.github.io/kyverno
helm install kyverno kyverno/kyverno -n kyverno --create-namespace
    Validation
    • A policy-violating pod is rejected at admission.

    Rollback: helm uninstall kyverno; policies no longer enforced.

  12. 12

    Workload validation

    low risk

    Prove the whole stack by scheduling a real GPU workload.

    • ·Run a CUDA test pod requesting a GPU.
    • ·Confirm it schedules onto a GPU node and sees the device.
    GPU test pod
    apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
    Validation
    • Pod completes and `nvidia-smi` lists the GPU.

    Rollback: `kubectl delete pod gpu-test` — no lasting change.

Generated artifacts are starting points — pin versions and review before running. Destructive steps are flagged; nothing here executes against a live cluster.