high riskUpgrades

GPU Operator upgrade

Upgrade the GPU Operator with a reviewed Helm diff and staged rollout.

Change scope

Cluster-wide operator (gpu-operator namespace)

Maintenance impact

Device-plugin/driver pods restart; brief GPU unavailability per node.

Prerequisites
  • ·Release notes reviewed (see KnowledgeOps)
  • ·Helm values captured
  • ·Backup of the current ClusterPolicy
Pre-checks
  • ·`helm diff upgrade` reviewed
  • ·Cluster healthy; no pending GPU pods
Procedure
  1. 1

    Review the change.

    bash
    helm repo update
helm diff upgrade gpu-operator nvidia/gpu-operator -n gpu-operator
  2. 2

    Upgrade the operator.

    bash
    helm upgrade --wait gpu-operator nvidia/gpu-operator -n gpu-operator
  3. 3

    Watch the device-plugin and driver DaemonSets roll out.

    bash
    kubectl -n gpu-operator rollout status ds/nvidia-device-plugin-daemonset
Validation
  • ✓All gpu-operator pods Running
  • ✓Nodes advertise `nvidia.com/gpu` allocatable
  • ✓Test GPU pod runs
Rollback

`helm rollback gpu-operator -n gpu-operator` to the prior revision; re-validate.

Communication

Announce brief per-node GPU unavailability during the rollout.