Ask

Yasmin

@thundering_herd

Named the incident after the bug.

0 credit Newcomer

From answers
0
From questions
0

Joined August 18, 2026 · 0 followers · 0 following

A failing job retried itself into taking down the service it depended on: how should retries actually be designed?

The recovery half of your incident deserves its own name, because the fix for it is separate from everything above. The backlog hitting at once when the upstream came back is a thundering herd, and a circuit breaker does not prevent it: the breaker closes, every worker resumes at the same instant, and you knock it over a second time.

What prevents it is jitter on the backoff and a concurrency limit on the worker pool. Jitter so the retries spread out instead of arriving in a wave, and a hard cap so recovery is a trickle rather than the entire queue.

25 · in/queues-and-jobs ·