AskMeAnything
Ask
471
@first_repo_finn ·

Pods stuck in Terminating for 20 minutes on every rollout of one deployment Troubleshooting

One deployment out of about forty takes roughly twenty minutes for old pods to disappear on every rollout. terminationGracePeriodSeconds is the default 30. The rest of the cluster rolls in seconds. kubectl describe on a stuck pod shows nothing obviously wrong, the containers show as not running, and the pod just sits in Terminating until it eventually goes. Node pressure is fine and the app itself has no long-running requests.

9 answers Share
Report

9 answers

BestTopNewOldControversial

Answering anonymously — a moderator will review it first.

  • @idempotent_ian · last mo. · 3 replies
    @idempotent_ian ·

    Containers not running but the pod still Terminating means the container runtime is done and something else is holding the pod object or the node's cleanup. Three things to check, in this order.

    Finalizers first: kubectl get pod <name> -o jsonpath='{.metadata.finalizers}'. If there is anything in there, some controller has claimed the right to clean up before deletion and is not finishing. A stale service mesh or a storage operator that was uninstalled badly are the usual sources.

    Then volume unmount. Check the kubelet logs on the node for that pod's UID — a hung NFS or CSI unmount will hold a pod in Terminating indefinitely and gives you almost nothing in describe. This is by far the most common cause of exactly twenty-ish minutes, because it lines up with unmount retry backoff.

    Third, sidecars. If a sidecar keeps running after the main container exits, the pod is not done, and twenty minutes is a suspiciously round number for something else's timeout.

    388
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @first_repo_finn · last mo.
      @first_repo_finn OP ·

      Finalizer from a mesh we removed two months ago, still on that one deployment's pods because the template was never updated. Removed it and rollouts are seconds now.

      104
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @pinned_versions · last mo.
      @pinned_versions ·

      Worth grepping the whole cluster for that finalizer while you are there. If one deployment kept it, others probably did too and just have not rolled yet.

      67
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @tabula_rasa_dev · last mo.
    @tabula_rasa_dev ·

    Separately worth checking your preStop hook and how the app handles SIGTERM, because the same symptom has a second common cause. If the entrypoint is sh -c "myapp", PID 1 is the shell and it does not forward SIGTERM, so the container waits out the full grace period and gets SIGKILLed every time. That gives you a consistent 30 second delay rather than 20 minutes though, so it is not your case here.

    142
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @pinned_versions · last mo. · 2 replies
    @pinned_versions ·

    The generic escape hatch is kubectl patch pod <name> -p '{"metadata":{"finalizers":null}}', but please treat that as a diagnostic rather than a fix. Whatever the finalizer was meant to clean up does not get cleaned up, so if it belongs to a live storage controller you can leak real resources. Use it to confirm the cause, then fix the cause.

    176
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @thermal_theo · last mo.
      @thermal_theo ·

      Learned this by leaking about forty orphaned volumes over a few weeks. Force deleting stuck pods became a habit and nobody noticed until the bill arrived.

      88
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @chainline_chris · last mo.
    @chainline_chris ·

    Once you have fixed it, add an alert on pods in Terminating for more than two minutes. It is a cheap signal that catches finalizer and unmount problems long before someone notices a slow rollout.

    118
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @late_stage_phd · last mo. · 2 replies
    @late_stage_phd ·

    Have you tried just increasing terminationGracePeriodSeconds? That usually sorts out slow shutdowns.

    21
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @idempotent_ian · last mo.
      @idempotent_ian ·

      That would make it worse rather than better here — the grace period is the maximum time before SIGKILL, so raising it lengthens the wait for anything that is genuinely hung. The containers are already stopped in this case, so the grace period is not the thing holding the pod.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report