high riskUpgrades

Kubernetes version upgrade

Upgrade the control plane, then GPU node pools, one minor version at a time, revalidating GPU enablement after.

Change scope

Control plane then node pools, one minor at a time

Maintenance impact

Control-plane brief disruption (HA mitigates); nodes drained in waves.

Prerequisites
  • ·Deprecated-API scan clean (e.g. pluto/kubent)
  • ·etcd backup taken
  • ·GPU Operator compatible with the target version
Pre-checks
  • ·`kubectl get nodes` all Ready
  • ·No deprecated APIs in use
Procedure
  1. 1

    Back up etcd, then upgrade the control plane (managed: node-pool image; kubeadm: `kubeadm upgrade`).

  2. 2

    Upgrade node pools in waves, draining before each.

    bash
    kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
  3. 3

    Re-validate GPU enablement (operator, device plugin) on upgraded nodes.

Validation
  • ✓Control plane + nodes at the target version
  • ✓GPU pods schedule and run
  • ✓Core workloads healthy
Rollback

Restore etcd from backup for the control plane; recreate node pools from the prior image. Kubernetes minor downgrades are not supported — roll forward or restore.

Communication

Multi-wave window; announce per-pool capacity reduction and any API deprecations affecting teams.