Your instinct about automatic retries is right, and the thing to do first is not to fix the flakes - it is to get them out of the way of the signal.
The damage is already done and you named it: a red build means nothing, so a real failure sat for two days. That is the actual cost, and it is much larger than the flakes themselves.
So, in order:
Identify them with data, not memory. Record every run: test name, outcome, commit, duration. Then a test that has failed and passed on the same commit is flaky by definition. You will find it is a small number of tests producing most of the noise: usually a handful.
Quarantine them. Move the known flaky ones into a separate non-blocking job. The main suite goes back to being trustworthy immediately, which is the whole point.
Give quarantine an expiry. A test that sits there forever is a deleted test with extra steps. Two weeks, with an owner, then fixed or deleted.
Then fix them properly, one at a time.
On automatic retries: they are acceptable around a quarantined set, and poisonous on the main suite. Retrying a failing test until it passes is a machine for hiding real intermittent bugs, and those are the expensive ones.