Eight workers pull jobs off a buffered channel (cap 100) and push results onto an unbuffered results channel. On SIGINT I call cancel() then wg.Wait(), and roughly one run in three it just sits there forever instead of exiting. Each worker does a select on ctx.Done() and the jobs channel, so I assumed cancellation was covered. Adding a counter shows two workers never return. What am I missing?
Separate bug you probably also have: when both cases of a select are ready, Go picks pseudo-randomly. So even after cancel, a worker with a full jobs channel keeps taking the jobs branch about half the time and never notices. If you want cancellation to win, check it first in its own non-blocking select before the real one.
This bit me on a batch importer with 40k queued items. Draining took eleven minutes after ctrl-c because the odds of hitting Done were roughly 50/50 each loop and there was always another job waiting.
Rewrite the whole thing on golang.org/x/sync/errgroup with errgroup.WithContext. You lose the manual WaitGroup, the first error cancels the shared context, and Wait returns it. I have not hand-rolled a pool in about four years and I do not miss it.
For diagnosis rather than the fix: next time it hangs, send it SIGQUIT instead of killing it. You get every goroutine's stack dumped to stderr, and the two stuck ones will both say chan send with the exact line. Beats adding print statements. GOTRACEBACK=all if you want the runtime ones too.
Put a time.Sleep(500 * time.Millisecond) between cancel and Wait. The goroutines need a moment to observe the cancellation and the scheduler doesn't always get to them in time.
That's not what's happening. A blocked channel send stays blocked for the heat death of the universe, not 500ms. Sleeping just moves the hang later.
@parquet_pile · 7mo ago · 3 replies
Your workers aren't stuck on the select, they're stuck on the send. Once cancel fires, whatever was draining the results channel exits, and any worker that already grabbed a job blocks forever on
results <- rbecause the receiver is gone.ctx.Done()does nothing for you there because you're not in a select at that point.Two fixes, pick one:
select { case results <- r: case <-ctx.Done(): return }The second is usually cleaner because it means results actually get flushed instead of silently dropped on shutdown.
Reply
Report
@oncall_omar · 7mo ago
That's exactly it. My consumer returns on
<-ctx.Done()in its own select, so it wins the race and then the workers are shouting into a closed room. Wrapping the send fixed it in 200 consecutive runs.Reply
Report
@blue_team_bex · 7mo ago
Worth adding the rule that saved me a lot of these: the goroutine that sends on a channel owns closing it, and the goroutine that receives never returns early unless it's also the one that told the senders to stop. Break either half and you get exactly this hang.
Reply
Report