Two minor versions behind on a managed cluster, about 40 nodes, roughly 200 deployments plus three StatefulSets with attached block storage. We have a four hour maintenance window on a Sunday and a small team. In-place upgrade of control plane then rolling node groups, or stand up a new cluster and cut over with DNS. I have done in-place twice on smaller clusters and never done a full swap.
Blue-green earns its keep when you are jumping several versions, changing something fundamental like the CNI, or you have stateless workloads and a proper GitOps setup that can rebuild the whole cluster from a repo. Two minor versions with StatefulSets is not that case. It is a good thing to build toward, but not a good thing to do for the first time under a deadline.
Whichever you pick, do it in a staging cluster first at the same versions, same add-ons. Not as a formality — I have caught a broken admission webhook and an operator that did not support the new API version that way, both of which would have eaten the entire window.
We have staging one version behind production, which I now realise is useless for this. Bringing it in line first.
Four hours is generous for two minor versions in place if you have prepared, and nowhere near enough for a first cluster swap. Whatever you plan, write down the abort criteria beforehand — at what time do you stop and roll back rather than pushing on because you are nearly there.
@pinned_versions · 11mo ago · 3 replies
In-place, and the StatefulSets are the deciding factor. Moving three sets of attached block storage to a new cluster means either snapshot and restore with real downtime, or replicating at the application layer, and neither is something you want to attempt for the first time inside a four hour window with a small team.
What I would actually do with that window: upgrade the control plane first and let it settle, then create new node groups on the new version alongside the old ones and cordon-and-drain the old ones gradually. That gives you a rollback that consists of un-cordoning the old nodes, which is worth more than anything else in this plan.
Before the window, run a deprecated API scan and check every admission webhook, CNI, CSI driver and operator against the target version. Almost every painful upgrade I have seen was an add-on, not the control plane.
Reply
Report
@idempotent_ian · 11mo ago
Also check your PodDisruptionBudgets before you drain anything. A budget that cannot be satisfied will stall a drain indefinitely and burn your window while nothing appears to be wrong.
Reply
Report
@chainline_chris · 11mo ago
The new-node-group-alongside pattern is the bit I was missing. That makes the rollback story much less frightening.
Reply
Report