Each user gets a digest email at 8am in their own timezone. Currently a single cron at midnight UTC loops the user table and enqueues 40k jobs, which pins the database for about twenty minutes and makes the app noticeably slow while it runs. I could bucket by timezone, or I could stop enqueueing entirely and just query for who is due every minute. Has anyone run this at this size and regretted their choice?
Stop enqueueing forty thousand rows. Run a small sweep every minute that selects the users whose local send time is now and who have not been sent today, and process that set. The queue table stays tiny, the load is spread evenly across the day instead of concentrated in one spike, and the whole thing is self healing because a missed minute is picked up by the next sweep rather than lost forever.
You need a sent-today marker with an index that the sweep can use, otherwise the sweep itself becomes the expensive query. That is the one thing to get right.
The self healing part is what sells it. Right now if the midnight cron fails, nobody gets anything and I find out from a customer.
Whichever route you take, the important thing is that sending is guarded by a per-user-per-day uniqueness constraint rather than by the enqueue being correct. I have watched a well intentioned retry on a dispatcher re-enqueue a whole page, and the only reason it did not send eight thousand duplicate digests is that there was a unique index on user and date that made the second attempt a no-op. Make the database refuse the duplicate. Do not rely on the scheduler being right.
We had a version of this and enqueued forty thousand jobs nightly for months without trouble, then shipped a bug where the digest builder threw for users with no activity. Every one of those jobs retried on the default policy, which turned forty thousand jobs into several hundred thousand attempts overnight and buried every other queue in the system, including password resets. The lesson was not about enqueue strategy at all. It was that a big batch and a generous retry policy are a bad combination, and that low priority bulk work needs its own queue and its own worker pool so it cannot starve anything that matters.
The pattern that fixed this for us was a dispatcher job that pages rather than a cron that loops. One job wakes up, selects a page of five hundred due users, enqueues those, and enqueues a copy of itself with the next cursor. The database sees a series of small bounded queries instead of one enormous transaction, the work is naturally resumable if a worker dies mid-run, and you can watch it progress in the dashboard.
We went from a twenty minute lock-heavy sweep to a stream of small jobs that finished in about four minutes, and the app stopped being slow during it. That was one afternoon of work and it did not require changing how anything downstream behaved.
Paging dispatcher is a good middle ground and much less invasive than what I suggested. The two approaches converge once you make the sweep incremental.
Sweep every minute for who is due. Unique index on user and date. Separate queue for bulk. Nothing has to happen at midnight.
@flakey_test · 7d ago
If you keep the enqueue model, at least bucket it. Twenty four crons, each handling one timezone offset, each enqueueing roughly a twenty fourth of your users an hour before their send time. Your peak becomes a couple of thousand rows instead of forty thousand, it runs in seconds rather than twenty minutes, and a failure affects one timezone rather than everybody. That is a half day change and it does not require rethinking the architecture, which matters if you are firefighting rather than redesigning.
Reply
Report