What is a good way to run background tasks in a web app — reports, emails, scheduled work?
My application needs to generate reports that take a minute or two, send emails, and run some scheduled maintenance overnight. Doing any of it inside a request is obviously wrong — the user waits and the request times out.
The options I can see range from a thread in the web process, through a proper task queue with a broker and workers, to just a scheduled script on the machine. They have very different amounts of moving parts and I cannot tell which complexity is justified.
How do you decide?
@queue_qamar · 4d ago
Decide on what happens when it fails, because that is the axis the options actually differ on.
A thread or a fire-and-forget task in the web process. The work dies with the process. A deploy, a crash or a restart loses it silently, and nothing tells you. Acceptable only when losing the task is genuinely fine — warming a cache, sending a nice-to-have notification.
A durable queue with separate workers. The task is written down before the request returns. It survives restarts, it retries on failure, it can be inspected, and failures land somewhere visible. This is what you want the moment a lost task means a customer does not get something they paid for.
A scheduled script. Simple and correct for genuinely periodic work with no per-request trigger. It is not a queue and does not become one; the moment you find yourself writing a table of pending work for the script to scan, you have written a worse queue.
So: is losing this task acceptable? That question sorts almost every case.
Reply
Report