Ask
211
@parquet_pile ·

Enqueue inside the database transaction or after commit, where do you put it?

Postgres with a Redis backed queue. If I enqueue inside the transaction, a rollback still leaves a job that goes looking for an order that does not exist. If I enqueue after commit, occasionally the process dies in the gap and the job is simply never created, which is worse because it is silent. I have also seen the worker pick the job up before the row is visible and fail on a not-found. Everyone seems to have an opinion here and I would like to hear from people running this in production rather than the theory.

11 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @two_vcpu_club · 6mo ago

    The race you described is real and faster than people expect. I instrumented ours after chasing exactly this bug: our workers were picking jobs off Redis in single digit milliseconds, and the transaction that created the row took a good deal longer than that under load because of a trigger nobody remembered writing. So the job genuinely arrived before the data existed, hundreds of times a day, and every one of those became a not-found in the logs that we had learned to ignore. The gap is not theoretical, and it gets worse under exactly the load where you least want mysterious failures.

    158
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @edge_runtime_bo · 6mo ago · 2 replies

    If your framework has an after-commit hook, use it, and understand what it does and does not promise. It removes the rollback problem completely, which is the failure that produces user visible nonsense like an email about a cancelled order. It does not remove the crash-in-the-gap problem, because nothing running in your process can. For a lot of applications that is an acceptable trade, and it is one line of code rather than a new component. Just be honest that you have chosen the silent-loss failure over the phantom-job failure rather than solved anything.

    134
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @pgpolicy_nadia · 6mo ago

      That is a fair characterisation and it is what most teams should do first. My objection is only to people believing the hook makes it correct.

      66
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @flakey_test · 6mo ago · 2 replies

    Going to disagree with the outbox recommendation for most teams, at least as a first move. Retrying on not-found with a short backoff solves the race, costs one line in the worker, and covers the overwhelming majority of real occurrences, because the gap is milliseconds and the retry lands after the commit. Combine that with an after-commit enqueue and you have handled both the rollback case and the visibility race for a fraction of the effort. Reach for an outbox when you can measure that you are losing jobs, not because a blog post says dual writes are unsound.

    121
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @two_vcpu_club · 6mo ago

      It works right up until you have a job whose absence nobody notices, which is exactly the class of bug the outbox exists for. Payments and emails, I would not do it. Cache invalidation, absolutely.

      74
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @pgpolicy_nadia · 6mo ago · 4 replies

    This is the dual write problem and there are only two honest solutions. Either the job lives in the same database as the data, so the enqueue is part of the same transaction and commits or rolls back with it, or you write an outbox row inside the transaction and a separate process moves outbox rows into Redis. Everything else is choosing which failure you would rather have.

    196
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @pgpolicy_nadia · 6mo ago

      The extra moving part is one small loop that selects unsent rows, pushes them, and marks them sent. It is genuinely small, and it is the only version of this that is correct rather than usually fine.

      91
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @flakey_test · 6mo ago

      And it gives you at-least-once delivery into Redis, which means your workers need to be idempotent anyway, which they should be regardless.

      52
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @parquet_pile · 6mo ago

      The outbox is the one I keep circling back to and keep talking myself out of because of the extra moving part.

      33
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @quietpackets · 6mo ago

    We enqueued inside the transaction for two years and it mostly worked, because rollbacks were rare. Then we added a validation step that failed late in the request, rollbacks stopped being rare, and for about a week we sent shipping notifications for orders that never existed. Customers were confused, support was busier than the bug deserved, and the fix took an hour. The real damage was that nobody trusted the notification system for months afterwards. Correct beats convenient on anything a customer sees.

    109
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @chmod_confused · 3h ago

    Same database as the data, or an outbox. After-commit if you accept the crash window. Never inside the transaction with an external queue.

    1
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report