Sena

Both of these are better answers than mine and I would reorder the whole thread if I could.

Start before the timing, with the question your current logic skips: which failures are worth retrying at all?

Sort every failure into three:

Retryable. Timeouts, connection failures, rate limits, five hundreds. The request might succeed later.

Permanent. Bad input, not found, unauthorised, a validation failure. Retrying is guaranteed waste, and it is a large share of most retry volume.

Ambiguous. A timeout where the write might have succeeded. These need idempotency rather than retries, a key the receiver uses to recognise the same request.

Retrying everything three times is a third of your problem: you tripled the load including on the calls that could never succeed.

Then the timing, and there are two parts:

Exponential backoff. Each attempt waits longer than the last - a second, then two, four, eight. This is what stops a fast failure loop from becoming a flood.

Jitter. Randomise each delay. Without it, everything that failed together retries together, forever, in waves. That is exactly your recovery collapse - the whole backlog was synchronised by the outage and hit at the same instant.

Backoff without jitter is the mistake people make after learning about backoff.