one duplicate row made my consumer send the same receipt 10 times Idempotency
Consumer inserts an invoice row, then emails a receipt. The payment provider redelivered invoice.paid once - which is their documented behaviour and entirely my problem to handle - my insert hit the unique constraint, the handler threw, the queue redelivered, and every single attempt sent another receipt before failing again on the same constraint.
max_retries is 10. One customer got ten identical receipts in about four minutes. She was extremely polite about it, which somehow made it worse.
Obviously the handler is wrong. What is the right shape?
@kombucha_kai · 4w ago · 3 replies
Let the database decide, and put the side effect after the write wins.
If no row comes back, this delivery is a duplicate and there is nothing to do. No exception, no retry, no second email.
The broader point: your retries are not the bug. At-least-once is the guarantee you bought when you picked a queue, and anything you put on it will run twice eventually - on redelivery, on a stalled worker, on a deploy. The handler has to be safe under that, always.
Reply
Report
@wick_and_wax · 4w ago
Key on the provider's event id, not the invoice id. Two different events can concern the same invoice and you want the second one to actually run. Deduping on the business object silently swallows real state changes and that bug takes months to find.
Reply
Report
@merino_max · 4w ago
Also check your batching. On several platforms a batch is acknowledged as a unit unless you ack messages individually, so one poisoned message drags nine healthy ones back through the retry with it. Drop the batch size to 1 while you are debugging this class of thing and the picture gets much clearer.
Reply
Report