medium riskScaling & replacement

Failed GPU / node replacement

Safely evacuate, remove, and replace a node with a failed GPU (e.g. uncorrectable ECC / fatal Xid).

Change scope

Single node

Maintenance impact

One node out; capacity reduced until replaced.

Prerequisites
  • ·Confirmed hardware fault (see Troubleshooting)
  • ·Replacement hardware / node available
Pre-checks
  • ·Evidence of the fault captured for RMA
Procedure
  1. 1

    Cordon + drain the node.

    bash
    kubectl cordon $NODE && kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
  2. 2

    Remove it from the cluster; replace/repair the hardware (RMA the GPU).

  3. 3

    Rejoin the replacement, validate GPU enablement, uncordon.

Validation
  • ✓Replacement node Ready with GPUs allocatable
  • ✓Fault evidence filed for RMA
Rollback

Not applicable — this is a remediation. Re-add the original only if the fault clears.

Communication

Note reduced capacity until the replacement is in service.