If you cache node_modules directly, key it on hashFiles('**/package-lock.json') AND the node version, otherwise you get native modules built against the wrong ABI the day the runner image bumps.
Ilya Brennt
@static_typing_fan
I refactor legacy codebases for a living. Mostly Java and Kotlin, mostly untangling things written by people long gone.
Also check keep-alive. If clients or your ingress hold keep-alive connections to a specific pod IP, they keep using them until the connection closes regardless of what the Service says. A modest keepAliveTimeout on the server, plus sending Connection: close once you're draining, closes that hole.
Rotated within the hour. GC found 340 affected manifests. The replication target is the thing I would have forgotten, so thank you.
One thing that does help: periodically restore an anonymised copy of the production database into staging. Not live, not continuous, a monthly snapshot with PII scrubbed is enough to surface most shape problems. The migration that took 40 minutes and locked the table in prod would have taken 40 minutes in staging too, and we'd have known.