go test -race ./... is clean on my laptop every single time. On CI it fails maybe one run in five, always in the same cache package, always the same two stacks. I have run the package alone locally about fifty times with no hit. I'd rather understand it than paper over it with a retry in the pipeline.
Worth saying plainly: the race exists on your laptop too. arm64 and amd64 have different memory models and different timing, so the window is narrower, but the bug is in the code, not in the CI machine. Don't spend three days on the repro when you could read the two stacks and see which field is shared.
Agreed. The race report names the exact memory address, the read line and the write line. In my experience that's enough to fix it without ever seeing it locally.
Audit your t.Parallel() calls in that package. A shared fixture built in TestMain, or a package-level testCache that two parallel subtests both write to, produces exactly this flakiness pattern. It fails more in CI because CI runs the whole suite and yours probably runs one package.
Make CI upload the failing output as an artifact rather than just showing the last 50 lines. The first stack is the read, the second is the write, and the second one is usually the one you never suspected.
@carbon_steel_kit · 5mo ago · 3 replies
The detector only reports races it actually observes, so "can't reproduce" mostly means "haven't hammered it enough". Try
go test -race -count=200 -cpu=1,2,4,8 ./internal/cache. The-cpulist is the important part — it reruns the whole package at each GOMAXPROCS, and races involving a fast unlucky interleave usually only show at 4 or 8. Your CI box probably has more cores than you think, or fewer.Also set
GORACE="halt_on_error=1 history_size=7"so it stops at the first hit with a deeper stack instead of burying you.Reply
Report
@blue_team_bex · 5mo ago
Hit it at count=200 with -cpu=8 on the third try. It's a map read in a metrics hook that runs on a timer. Thanks.
Reply
Report
@oncall_omar · 5mo ago
history_size=7 costs a lot of memory but it's the difference between a useful second stack and "goroutine created at some point, good luck".
Reply
Report