Ask
86
@wick_and_wax ·

is a dead letter queue overkill for a side project doing 200 jobs a day Dead Letter

Side project. About 200 jobs a day, mostly outbound webhooks to customers' own endpoints. Every architecture diagram I read has a dead letter queue in it and I cannot tell whether that is for me or for people doing 200,000 a day.

Right now a failure retries three times and then it is gone - there is a log line somewhere with a seven day retention. That does feel bad.

Is a DLQ what fixes that or am I copying a diagram?

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @nightbus_nina · 4mo ago · 3 replies

    Mostly copying a diagram, but the discomfort is correct and worth acting on.

    A dead letter queue gives you two things: somewhere failed messages land instead of vanishing, and a mechanism to replay them. At 200 a day you need the first and you can do the second by hand.

    A failed_jobs table with payload, error, attempt count and replayed_at gets you there in about thirty lines, and you can query it, join it to the customer, and put it on a page - none of which a platform DLQ lets you do comfortably.

    The thing people actually get wrong: a DLQ is not an alert. Messages arrive silently and sit there for months. Whatever you build, the part that matters is that something tells you.

    152
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @r2_bucketeer · 4mo ago

      "A DLQ is not an alert" should be on a poster. I have inherited two systems with tens of thousands of messages in a dead letter queue that nobody had ever opened, in one case including the entire first week of a paying customer's data.

      49
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @fg_stall · 4mo ago

      The replay button always arrives later and always as an emergency. Design the table now so that replay is one insert back into the main queue, not a script you write at midnight.

      17
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @merino_max · 4mo ago · 2 replies

    For outbound webhooks specifically, failure is not exceptional - it is Tuesday. Customer endpoints go down for hours at a time and it is not your fault or theirs.

    What you want:

    • retry with real backoff over hours, not three attempts over a minute
    • a per-endpoint circuit breaker so one dead customer cannot occupy all your workers
    • a page where the customer can see their own failed deliveries, with the response body, and press resend themselves

    That last one deletes most of your future support load and is the only one of the three that anybody thanks you for.

    79
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @hollis_pike · 4mo ago

      The circuit breaker matters more than it sounds at 200 a day. One customer whose endpoint hangs for 30 seconds instead of refusing the connection will eat every worker you own, and from the outside it looks like your queue is broken rather than theirs. Aggressive timeouts on outbound calls, always.

      28
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @adjunct_life · 4mo ago

    The log line is completely fine right up to the day you need to know which six of two hundred failed and your retention expired on Thursday.

    19
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @thermal_theo · 4mo ago

    If your platform gives you a dead letter queue for three lines of config, take it - the cost is near zero and failed messages stop being log lines with a seven day life. Just wire the alert on non-empty at the same time, and go and look at it weekly until you trust it.

    41
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @thriftedtweed · 4mo ago

    Store the response body from their endpoint, truncated. "It failed" starts an investigation. "It returned 413 with a body about payload size" ends the conversation in one email.

    8
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report