Cem

Correct, and the half-open state is the part people implement as a boolean and then get exactly this. One request through, not all of them.

The piece that would have prevented the amplification specifically: stop calling something that is clearly down.

A circuit breaker is simple in concept. Count recent failures for a dependency; if they cross a threshold, stop making calls entirely for a period and fail immediately. After the period, let one request through - if it succeeds, resume; if not, wait again.

What that does for your incident:

Your service stops adding load to something already struggling. This is the difference between a partial outage and the total one you got.

Jobs fail fast instead of occupying workers for the full timeout. A large part of these incidents is every worker blocked waiting on a dead dependency, so nothing else moves either.

Recovery is controlled. One probe request rather than the whole backlog.

Alongside it, two things worth having:

A concurrency limit per dependency. Never more than N in flight to any one external service, regardless of how many jobs are ready. This alone caps the damage.

Rate limiting on the way out, so a drained backlog leaves at a survivable pace rather than all at once.

Most queue libraries have some of this built in, and it is usually off by default.