Rolling update in Kubernetes, new replicas go Ready quickly, but for roughly 40 seconds after the rollout reports complete we still see responses from the old version, confirmed by a version header we stamp on every response. terminationGracePeriodSeconds is 30, we send SIGTERM and the app exits in about 2 seconds. Where are the extra requests coming from?
Also check keep-alive. If clients or your ingress hold keep-alive connections to a specific pod IP, they keep using them until the connection closes regardless of what the Service says. A modest keepAliveTimeout on the server, plus sending Connection: close once you're draining, closes that hole.
Do you have an external LB with its own health check interval? Ours was 10s interval, 3 failures to mark unhealthy, so 30 seconds of routing to a dead backend, which is almost exactly the tail you're describing. Tightened to 2s and 2 failures and the mystery went away.
Ours is 5s and 2, so probably not the main cause, but I hadn't even thought to look. Checking.
Small thing, but confirm the version header is stamped per request and not baked into something cached upstream. I chased a ghost old version for a day and it was a CDN holding a status page.
@rollback_rae · 5mo ago · 3 replies
Endpoint propagation. When a pod goes Terminating, kube-proxy and your ingress controller find out asynchronously, and there is a window where the Service still lists it. Your app exiting in 2 seconds is actually the bug: it dies while load balancers are still handing it work.
Standard fix is a preStop hook that sleeps 5-15 seconds before the app starts shutting down, so the pod stops being advertised, in-flight requests finish, then you exit. Also make readiness start failing on SIGTERM, not just liveness.
The mental model that fixed this for me: readiness controls whether you get new work, SIGTERM handling controls whether you finish the work you already have. Two separate switches, and they have to flip in that order.
Reply
Report
@cors_error_cleo · 5mo ago
Worth putting the sleep inside the app's own signal handler rather than relying on
sleepexisting in a distroless base image. Bit me once and the failure is completely silent.Reply
Report
@hot_heap_hana · 5mo ago
preStop sleep 10 plus failing readiness on SIGTERM took the tail from 40s to about 3s. I had it exactly backwards, I'd spent a week trying to make shutdown faster.
Reply
Report