The suite fails about one run in ten and passes on retry, how do teams actually deal with flaky tests?

For actually finding the cause, the categories are few and you can usually identify which one you have quickly.

Shared state between tests. The most common by a distance. A database row, a file, an environment variable, a module-level cache, a clock somebody stubbed and did not restore. Symptom: the test passes alone and fails in the suite, or fails only in a particular order.

Test order dependence. Related, and easy to prove - run the suite with randomised order locally. If failures appear, you have it. A lot of test runners can do this with a flag and it is the single most productive ten minutes here.

Real timing and waiting. Sleeps instead of waiting for a condition, and anything asserting on something that has not happened yet. Symptom: fails more on slow or loaded machines, which is why it fails in CI and not on your laptop.

Real dependencies. A test hitting the network or a live service. Intermittent by nature.

Resource limits. CI runners are usually smaller than a developer machine, so memory or parallelism issues show up only there.

The reason you cannot reproduce locally is nearly always the last two, your laptop is faster and less contended. Try running the suite with reduced parallelism on a loaded machine and a fair number will suddenly be reproducible.

Your instinct about automatic retries is right, and the thing to do first is not to fix the flakes - it is to get them out of the way of the signal.

The damage is already done and you named it: a red build means nothing, so a real failure sat for two days. That is the actual cost, and it is much larger than the flakes themselves.

So, in order:

Identify them with data, not memory. Record every run: test name, outcome, commit, duration. Then a test that has failed and passed on the same commit is flaky by definition. You will find it is a small number of tests producing most of the noise: usually a handful.

Quarantine them. Move the known flaky ones into a separate non-blocking job. The main suite goes back to being trustworthy immediately, which is the whole point.

Give quarantine an expiry. A test that sits there forever is a deleted test with extra steps. Two weeks, with an owner, then fixed or deleted.

Then fix them properly, one at a time.

On automatic retries: they are acceptable around a quarantined set, and poisonous on the main suite. Retrying a failing test until it passes is a machine for hiding real intermittent bugs, and those are the expensive ones.

One thing worth adding, because it is the part people find hardest to accept: some flaky tests are correct and the code is flaky.

A test that intermittently fails on a race condition is doing its job. It has found a real bug that happens rarely, which is exactly the kind you want found before a user does. Deleting it or retrying it away removes your only detection of a genuine defect.

So when you triage each one, the first question is not how do I stabilise this test: it is could this failure ever happen in production? If yes, the test stays and the code gets fixed.

How to tell them apart, roughly:

• Test-only flakiness involves test infrastructure, fixtures, ordering, cleanup, timing of assertions.
• Real flakiness involves the system under test - concurrency, retries, ordering of real operations, anything with a clock or a queue in it.

The second category is worth a lot of attention and it is systematically under-investigated, because the easy fix is to make the test wait longer and everybody feels better.

And whatever you do, keep the failure logs. The hardest part of fixing a flake is that by the time somebody looks, the run has been retried and the evidence is gone.