A failing job retried itself into taking down the service it depended on: how should retries actually be designed?

The piece that would have prevented the amplification specifically: stop calling something that is clearly down.

A circuit breaker is simple in concept. Count recent failures for a dependency; if they cross a threshold, stop making calls entirely for a period and fail immediately. After the period, let one request through - if it succeeds, resume; if not, wait again.

What that does for your incident:

Your service stops adding load to something already struggling. This is the difference between a partial outage and the total one you got.

Jobs fail fast instead of occupying workers for the full timeout. A large part of these incidents is every worker blocked waiting on a dead dependency, so nothing else moves either.

Recovery is controlled. One probe request rather than the whole backlog.

Alongside it, two things worth having:

A concurrency limit per dependency. Never more than N in flight to any one external service, regardless of how many jobs are ready. This alone caps the damage.

Rate limiting on the way out, so a drained backlog leaves at a survivable pace rather than all at once.

Most queue libraries have some of this built in, and it is usually off by default.

Start before the timing, with the question your current logic skips: which failures are worth retrying at all?

Sort every failure into three:

Retryable. Timeouts, connection failures, rate limits, five hundreds. The request might succeed later.

Permanent. Bad input, not found, unauthorised, a validation failure. Retrying is guaranteed waste, and it is a large share of most retry volume.

Ambiguous. A timeout where the write might have succeeded. These need idempotency rather than retries, a key the receiver uses to recognise the same request.

Retrying everything three times is a third of your problem: you tripled the load including on the calls that could never succeed.

Then the timing, and there are two parts:

Exponential backoff. Each attempt waits longer than the last - a second, then two, four, eight. This is what stops a fast failure loop from becoming a flood.

Jitter. Randomise each delay. Without it, everything that failed together retries together, forever, in waves. That is exactly your recovery collapse - the whole backlog was synchronised by the outage and hit at the same instant.

Backoff without jitter is the mistake people make after learning about backoff.

Fixed three attempts five seconds apart is in every tutorial and it is wrong in every one of them.

Two operational details that matter as much as the algorithm.

Cap the total attempts and send failures somewhere. After the final attempt, the job goes to a dead letter queue rather than being retried forever or dropped. Then a person can look at what failed and requeue deliberately when the dependency is genuinely healthy. Draining a dead letter queue on purpose is completely different from a backlog releasing itself.

Make retries visible. A metric for retry count per dependency, and an alert when it rises sharply. In your incident the retry rate would have gone up minutes before anything else looked wrong, and it is the earliest signal you can get. Most people only alert on failures, which fires later.

On the specific shape I would use for your case: exponential backoff with full jitter, a small number of attempts for genuinely transient failures, no retries at all for permanent ones, a concurrency cap on the upstream, and a breaker.

And one thing to check while you are in there: make sure retries are not nested. An HTTP client that retries three times, inside a job that retries three times, is nine calls per job and nobody planned it. Layered retries are how a modest policy becomes an accidental flood, and they are very easy to acquire without noticing.