Once you have fixed it, add an alert on pods in Terminating for more than two minutes. It is a cheap signal that catches finalizer and unmount problems long before someone notices a slow rollout.
Chris
@chainline_chris
Home mechanic for fifteen years; owns too many tools and regrets almost none of them.
Two on the same morning is worth quietly investigating rather than filing under bad luck. In my experience it is occasionally genuinely coincidence, and it is occasionally two people who both did not want to work that shift and coordinated. Look at whether it was the same pairing, whether either had asked for that day off, and whether Saturday is under-rewarded compared with your quieter shifts.
Worth confirming whether the failing pods are only the newly scheduled ones or also existing pods on old nodes. If existing pods are fine, it is startup ordering on the new node. If everyone gets it, CoreDNS itself is saturating and the replica count is the story.
Also worth trimming hours rather than the menu if the drop is footfall. Look at your takings by hour for the last three weeks — the first and last hour of the day are often costing you more in wages than they bring in during January, and closing an hour early for six weeks is invisible to most customers.
There is a variation clause with 14 days notice, which they have technically met. Going in with the phased ask instead.
This is the practical answer I was missing. I had framed it as reprint or absorb and there is obviously a middle.
Third option nobody has said yet: hire the admin out, not the chair. A bookkeeper for a few hours a month, and moving bookings to something that handles its own reminders, took roughly eight hours a week off me for a fraction of a part-timer's cost. Your 55 hours is not all chair time.
The new-node-group-alongside pattern is the bit I was missing. That makes the rollback story much less frightening.
We have staging one version behind production, which I now realise is useless for this. Bringing it in line first.
For the arithmetic: 300 a month plus 9k over, say, five years is roughly 450 a month of cost. At a typical hourly rate you probably need somewhere around six to ten billed hours a week just to break even on it, and you are at eight. So the room is not losing money dramatically, it is simply not earning yet. That is a different problem from a mistake.