Operate

Day-Two Operations

Governed runbooks for the ongoing work of running a GPU platform — upgrades, patching, rotation, scaling, and disaster recovery. Every procedure has pre-checks, validation, and a rollback.

Upgrades

high risk

GPU driver upgrade

Upgrade the NVIDIA/AMD driver on GPU nodes, one at a time, via the GPU Operator, validating before moving on.

Open runbook high risk

GPU Operator upgrade

Upgrade the GPU Operator with a reviewed Helm diff and staged rollout.

Open runbook high risk

Kubernetes version upgrade

Upgrade the control plane, then GPU node pools, one minor version at a time, revalidating GPU enablement after.

Open runbook

Patching

medium risk

Kernel / OS security patch

Apply OS/kernel security patches with a rolling reboot, keeping the driver aligned to the kernel.

Open runbook

Rotation

medium risk

Certificate rotation

Rotate cluster certificates ahead of expiry and automate future rotation.

Open runbook medium risk

Secret rotation

Rotate secrets in the vault and roll them out via the External Secrets Operator.

Open runbook

Scaling & replacement

medium risk

Failed GPU / node replacement

Safely evacuate, remove, and replace a node with a failed GPU (e.g. uncorrectable ECC / fatal Xid).

Open runbook low risk

Cluster scaling (add GPU capacity)

Add GPU nodes to absorb growing demand, validating enablement before admitting workloads.

Open runbook

Backup & DR

low risk

Backup & disaster-recovery test

Rehearse restoring the platform and workloads from backups to prove RPO/RTO.

Open runbook