Ask
167
@two_vcpu_club ·

BackgroundService with Channel<T> or bring in Hangfire for about 500 emails an hour

One ASP.NET Core app on a single box with Postgres already there, deploys restart the process two or three times a week. Right now sending is fire and forget with Task.Run from the controller, which I know is wrong, and I have already lost mail to a restart. The two options in front of me are an in-process channel consumed by a hosted service, or pulling in a real job library with a dashboard. What did people with roughly this volume actually end up running?

11 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @pgpolicy_nadia · 2mo ago

    Hangfire with the Postgres storage provider is about the smallest amount of work that gets you durability, retries and a dashboard you will actually use at three in the morning. It is a package, a connection string and a couple of lines in startup, and the dashboard alone has justified it for me twice, because being able to see a failed job with its exception and press retry beats writing your own admin screen.

    The cost is that it creates and polls its own tables, which on a small box is a background load you should be aware of but which at your volume is irrelevant. Five hundred an hour is roughly one every seven seconds. That is nothing.

    149
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @flakey_test · 2mo ago · 4 replies

    Channel<T> is in memory, so a deploy loses whatever is queued, which is precisely the failure you already had. Five hundred an hour is nothing, so throughput is not your problem. Durability is. Pick whichever option writes the job to Postgres before the request returns.

    172
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @pgpolicy_nadia · 2mo ago

      And with graceful shutdown you can drain a channel on stop, which helps with planned deploys and does nothing at all for a crash or an OOM kill.

      48
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @flakey_test · 2mo ago

      Right. Channels are excellent for work you can cheaply redo, like cache warming or metrics. Not for anything a customer is waiting to receive.

      74
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @two_vcpu_club · 2mo ago

      So the channel only makes sense if losing an item is genuinely acceptable.

      26
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @parquet_pile · 2mo ago · 3 replies

    I would push back on adding a library at all for this. An outbox table with a status column and a hosted service that polls it every few seconds is genuinely about a hundred lines including the retry logic, and you already have Postgres. You get durability across restarts, you can query the queue with SQL you already know, and you own the whole thing.

    The reason I feel strongly about it is that job libraries bring their own schema, their own upgrade path and their own opinions about serialisation, and I have been burned by a version upgrade rewriting job storage on a system I did not want to think about. For one queue with one job type, the library is more moving parts than the problem.

    131
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @pgpolicy_nadia · 2mo ago

      It is a hundred lines until you want scheduled jobs, exponential backoff, a dead letter path and visibility into what failed. Then it is a thousand lines and it is your job library, just less tested than the one you did not install.

      88
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @parquet_pile · 2mo ago

      Fair, and that is the real question: does the app grow more job types? For one email queue I stand by the hundred lines. For five job types with schedules I would install the library too.

      57
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @edge_runtime_bo · 2mo ago

    Whichever you choose, make the send idempotent before you make it durable. Both approaches will eventually run a job twice, because at-least-once is what you get from anything that retries, and a duplicate email is worse than a slightly delayed one. Store a key on the row for what the message is about, check it before sending, and record the send in the same transaction that marks the job done. That single decision has saved me far more embarrassment than any queue choice.

    117
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @chmod_confused · 2mo ago

    Table in Postgres, hosted service polling it, idempotency key on the row. Reach for a library when you have a third job type or a schedule.

    84
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @quietpackets · 2mo ago

    My version of your current setup was Task.Run inside a controller action on an app hosted behind IIS. It worked perfectly for months. Then the app pool recycled during a quiet period and swallowed about forty confirmation emails with no error anywhere, because the exception went to a task nobody was awaiting. We only found out when customers rang. The thing that actually hurt was not the lost mail, it was that we had no record that the work had ever been requested, so we could not replay it. Write the intent down first, in the database, and everything after that is recoverable.

    103
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report