Drained a node for maintenance. One StatefulSet pod will not reschedule — its PVC sits in Pending and the pod is Pending with FailedScheduling. The PV it used shows status Available in kubectl get pv. Storage class is the cloud provider default with WaitForFirstConsumer. Other pods from the same StatefulSet came back fine on other nodes. Cluster spans three zones.
kubectl describe pod on the Pending pod usually says this out loud in the events — something like "node(s) had volume node affinity conflict". It is one of the more helpful messages Kubernetes produces and it is easy to skip past when you are looking at the PVC instead of the pod.
It did say exactly that and I had read it three times without understanding what it meant. Obvious in hindsight.
If this is a workload you want to survive losing a zone, the storage choice is the real decision rather than the drain procedure. Zonal block storage plus a single-replica StatefulSet is a single point of failure by design. Either accept that and plan capacity per zone, or move to something replicated at the storage layer.
Small operational note: drain one node at a time and wait for everything to reschedule before the next, especially with StatefulSets. I have seen someone drain a whole zone's worth in parallel because the nodes came back healthy, then spend an afternoon on this.
@idempotent_ian · 7mo ago · 3 replies
Almost certainly a zone mismatch. Cloud block volumes are tied to the zone they were created in, and the PV carries that as node affinity. If the node you drained was the last schedulable node in that zone, or the remaining capacity in that zone is full, the scheduler has nowhere to put a pod that must reach that volume, and the PVC sits Pending forever.
Check it with
kubectl get pv <name> -o yamland look atspec.nodeAffinity— it will name a zone. Then compare against which zones have schedulable nodes with room.Also check the PV's
claimRef. An Available PV that still has a claimRef pointing at a deleted PVC will never bind to a new one, which produces the same Pending symptom for a completely different reason.Reply
Report
@pinned_versions · 7mo ago
Worth adding a pod disruption budget and checking per-zone headroom before draining anything that has attached storage. This failure mode is entirely predictable and almost always discovered during maintenance.
Reply
Report
@fiber_finn · 7mo ago
Zone c had exactly two nodes and I drained one of them, the other was full. Added capacity in that zone and it bound within seconds.
Reply
Report