One thing worth adding, because it is the part people find hardest to accept: some flaky tests are correct and the code is flaky.
A test that intermittently fails on a race condition is doing its job. It has found a real bug that happens rarely, which is exactly the kind you want found before a user does. Deleting it or retrying it away removes your only detection of a genuine defect.
So when you triage each one, the first question is not how do I stabilise this test: it is could this failure ever happen in production? If yes, the test stays and the code gets fixed.
How to tell them apart, roughly:
- Test-only flakiness involves test infrastructure, fixtures, ordering, cleanup, timing of assertions.
- Real flakiness involves the system under test - concurrency, retries, ordering of real operations, anything with a clock or a queue in it.
The second category is worth a lot of attention and it is systematically under-investigated, because the easy fix is to make the test wait longer and everybody feels better.
And whatever you do, keep the failure logs. The hardest part of fixing a flake is that by the time somebody looks, the run has been retried and the evidence is gone.