Every helm upgrade takes ten minutes longer than it used to, but still exits 0 — where is the time going?
Our deploy job used to finish in about two minutes. Since some point in the last few weeks the same job takes twelve to fifteen, on the same chart against the same cluster.
What makes it strange is that nothing fails. No error, no warning, exit code 0, release reports deployed, and the application comes up correctly at the end. It is purely slower.
Watching the output, it sits for a very long time on a wait — and the resource it claims to be waiting for is already gone if I check with kubectl from another terminal while it hangs. It sits there anyway, and then carries on as though nothing happened.
The chart has a few hooks with a delete policy that removes the previous hook object before creating the new one, so there are a handful of these waits per upgrade. Each one seems to burn the full clock.
I cannot find anything in our diff that would cause this, and because it exits 0 nothing alerted. What would you check?
@exit_code_zero · 2h ago
Check your Helm version before you look at anything else. This is a known regression in 4.2.1 and your description matches it exactly, including the part that makes it so hard to spot.
What it does. Any wait for a resource to be deleted stops returning when the resource is actually gone and instead runs for the whole
--timeout. The object disappears in milliseconds, the wait sits there for the full clock, and then continues. Nothing errors, the release reports success, and the exit code is 0.Why your chart is affected and somebody else's is not. You said it: hooks with
helm.sh/hook-delete-policy: before-hook-creation. Every upgrade deletes the previous hook object and waits for it to be gone before recreating it, so each hook costs you one full timeout. That is why the damage scales with the number of hooks rather than the size of the release, and why a chart with no hooks looks fine.The fix is a version bump. 4.2.2 reverts the change that caused it. If you cannot move forward for some reason, 4.2.0 is the last version before it and is what people rolled back to.
The thing worth taking away is why this survived long enough to reach you. A wait that gives up and continues is a wait that fails open: the only symptom it can produce is time. No exit code, no log line at a level anybody alerts on, no failed pod. Deploy duration is not usually on a dashboard, so the one signal it emits is the one nobody is watching. Worth adding — a job that doubles in length is telling you something even when it succeeds.
Reply
Report