Ask
26
@edge_ekin ·

What is a good way to run background tasks in a web app — reports, emails, scheduled work?

My application needs to generate reports that take a minute or two, send emails, and run some scheduled maintenance overnight. Doing any of it inside a request is obviously wrong — the user waits and the request times out.

The options I can see range from a thread in the web process, through a proper task queue with a broker and workers, to just a scheduled script on the machine. They have very different amounts of moving parts and I cannot tell which complexity is justified.

How do you decide?

4 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @queue_qamar · 4d ago

    Decide on what happens when it fails, because that is the axis the options actually differ on.

    A thread or a fire-and-forget task in the web process. The work dies with the process. A deploy, a crash or a restart loses it silently, and nothing tells you. Acceptable only when losing the task is genuinely fine — warming a cache, sending a nice-to-have notification.

    A durable queue with separate workers. The task is written down before the request returns. It survives restarts, it retries on failure, it can be inspected, and failures land somewhere visible. This is what you want the moment a lost task means a customer does not get something they paid for.

    A scheduled script. Simple and correct for genuinely periodic work with no per-request trigger. It is not a queue and does not become one; the moment you find yourself writing a table of pending work for the script to scan, you have written a worse queue.

    So: is losing this task acceptable? That question sorts almost every case.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @queue_qamar · 4d ago

    The complexity worry is fair, and there is a middle option people miss: use your database as the queue.

    A table of jobs, a worker process that claims rows with a conditional update, and a retry column. No broker, no new infrastructure, transactional with the rest of your data — which is a genuine advantage, because enqueueing a job in the same transaction as the change that caused it removes an entire class of bug where one happens and the other does not.

    Several frameworks now ship exactly this as a built-in, precisely because most applications never needed the broker.

    When you outgrow it: very high throughput, or fan-out to many consumers. That threshold is far higher than people assume. A database-backed queue handles a workload most applications will never reach, and the operational simplicity is worth a lot.

    27
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @acme_asu · 2d ago

    One thing to settle early because it changes the design: does the user need to see the result?

    If a report is generated and then emailed, fire the job and forget it. If the user is watching a page waiting for it, you also need a way to report progress and completion back — a status the page polls, or a push channel.

    That requirement is what usually drives people from a scheduled script to a real queue, more than throughput ever does, because a script gives you nowhere to put the status.

    15
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @pipeline_pars · 2d ago

    Whichever you choose, the things that make it survivable in production:

    • Make jobs idempotent. Every system will run a job twice eventually — a retry after a timeout that actually succeeded is the classic. Design so that running it twice is harmless, and this whole category of incident disappears.
    • Pass identifiers, not objects. Store a record ID and re-read it in the worker. A serialised object is a snapshot that is already stale by the time it runs, and it breaks when the class changes under it.
    • Separate the queues. Fast and slow work in one queue means a batch of reports blocks every password reset email.
    • Alert on queue depth and on age of the oldest job. Depth alone lies; a queue of ten jobs that have been waiting an hour is a worse signal than a thousand moving quickly.
    • Have a visible dead-letter place for what failed permanently. Jobs that vanish after their last retry are the ones you find out about from a customer.
    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report