Ask
157
@thermal_theo ·

one duplicate row made my consumer send the same receipt 10 times Idempotency

Consumer inserts an invoice row, then emails a receipt. The payment provider redelivered invoice.paid once - which is their documented behaviour and entirely my problem to handle - my insert hit the unique constraint, the handler threw, the queue redelivered, and every single attempt sent another receipt before failing again on the same constraint.

max_retries is 10. One customer got ten identical receipts in about four minutes. She was extremely polite about it, which somehow made it worse.

Obviously the handler is wrong. What is the right shape?

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @kombucha_kai · 4w ago · 3 replies

    Let the database decide, and put the side effect after the write wins.

    const [row] = await db
      .insert(invoices)
      .values({ providerEventId: event.id, ... })
      .onConflictDoNothing({ target: invoices.providerEventId })
      .returning({ id: invoices.id })
    
    if (!row) return   // someone already handled this event, ack and go home
    
    await sendReceipt(row.id)
    

    If no row comes back, this delivery is a duplicate and there is nothing to do. No exception, no retry, no second email.

    The broader point: your retries are not the bug. At-least-once is the guarantee you bought when you picked a queue, and anything you put on it will run twice eventually - on redelivery, on a stalled worker, on a deploy. The handler has to be safe under that, always.

    184
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @wick_and_wax · 4w ago

      Key on the provider's event id, not the invoice id. Two different events can concern the same invoice and you want the second one to actually run. Deduping on the business object silently swallows real state changes and that bug takes months to find.

      56
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @merino_max · 4w ago

      Also check your batching. On several platforms a batch is acknowledged as a unit unless you ack messages individually, so one poisoned message drags nine healthy ones back through the retry with it. Drop the batch size to 1 while you are debugging this class of thing and the picture gets much clearer.

      26
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @finops_reyna · 4w ago

    Separate "record that it happened" from "do the thing".

    Handler one writes the fact in a transaction and nothing else. A second step - an outbox row picked up by another job - sends the email. Then a mail provider outage does not re-run your accounting, and a constraint violation in accounting does not send anything at all.

    It is one more moving part and it is worth it the first time your email provider has a bad afternoon.

    103
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @blown_cap · 4w ago

    Generalise it so you stop thinking about it per-handler: every job carries an idempotency key from the producer, and you have a processed_jobs(key primary key, completed_at) table. First statement in the handler inserts the key, in the same transaction as the work. Conflict means you are a duplicate, return.

    Boring, portable across every queue you will ever use, and about fifteen lines of shared code.

    49
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @shopvac_ghost · 4w ago · 2 replies

    Set max_retries to 1. Worst case a duplicate sends twice instead of ten times and you have capped the blast radius.

    28
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @nightbus_nina · 4w ago

      And now a genuine transient failure - thirty seconds of database unavailability, a redeploy mid-batch - silently drops the event, and you find out at month end when the numbers do not tie. That is a much worse bug than ten emails, because nobody reports it.

      Retries protect you from the world. Idempotency protects you from retries. They are not alternatives, you need both.

      19
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @hollis_pike · 4w ago

    Log the event id and the attempt number on every single run, structured. Next time this happens - and it will, in a different handler - you can answer "was that ten events or one event ten times" in twenty seconds instead of an afternoon of reading mail provider logs.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report