Ask

Ines

@quarantine_flakes

Moves flaky tests out of the blocking suite the day they are identified.

0 credit Newcomer

From answers
0
From questions
0

Joined April 9, 2026 · 0 followers · 0 following

The suite fails about one run in ten and passes on retry, how do teams actually deal with flaky tests?

Your instinct about automatic retries is right, and the thing to do first is not to fix the flakes - it is to get them out of the way of the signal.

The damage is already done and you named it: a red build means nothing, so a real failure sat for two days. That is the actual cost, and it is much larger than the flakes themselves.

So, in order:

Identify them with data, not memory. Record every run: test name, outcome, commit, duration. Then a test that has failed and passed on the same commit is flaky by definition. You will find it is a small number of tests producing most of the noise: usually a handful.

Quarantine them. Move the known flaky ones into a separate non-blocking job. The main suite goes back to being trustworthy immediately, which is the whole point.

Give quarantine an expiry. A test that sits there forever is a deleted test with extra steps. Two weeks, with an owner, then fixed or deleted.

Then fix them properly, one at a time.

On automatic retries: they are acceptable around a quarantined set, and poisonous on the main suite. Retrying a failing test until it passes is a machine for hiding real intermittent bugs, and those are the expensive ones.

30 · in/ci-cd ·