One $12 VPS, Postgres on the same box, a Node app. About 20 jobs a minute: thumbnails, transactional email, a nightly export. No spikes worth the name.
Everything I read says use Redis. Redis means another process to keep alive, another thing to decide whether to back up, and another failure mode at 3am on a machine with 2 GB of RAM.
Is there a real reason not to just use the database that is already there?
Use a maintained library rather than hand-rolling the table. The naive version is about forty lines and the missing four hundred are scheduling, retry with backoff, visibility timeouts, archiving completed rows, and the migrations to change any of that later without downtime.
Every hand-rolled queue I have inherited was correct for the first six months and then grew a WHERE status = 'stuck_weird' clause.
Watch the dead tuples. A queue table is the highest-churn table you will ever own, and default autovacuum settings are tuned for tables that are mostly read. Set aggressive autovacuum on that table specifically and archive completed rows out to a history table.
Otherwise you are back here in six months asking why a SELECT over 3,000 live rows takes 400ms, and the answer will be four million dead ones.
autovacuum_vacuum_scale_factor = 0.01 on that one table is the usual starting point, with the threshold lowered to match. Per-table, not globally.
One box means one restart takes everything down anyway. Adding Redis there does not buy availability, it buys a second daemon competing for 2 GB of RAM behind the same single point of failure.
Postgres queues fall over somewhere around 100 jobs a second. It is a known ceiling and you will hit it eventually and regret the whole thing.
There is no 100 a second ceiling. Published benchmarks of skip-locked queues on modest hardware sit in the thousands per second, and the practical limits that do exist come from connection count and vacuum behaviour, both of which are configuration rather than architecture.
Also OP is asking about 0.33 jobs per second. This is refusing to drive to the shops because the car tops out below the speed of sound.
@spreadsheet_tabs · 8mo ago · 3 replies
No, and the performance argument everyone reaches for is the wrong argument anyway.
FOR UPDATE SKIP LOCKEDis what every Postgres queue is built on and it will do thousands of jobs a second on hardware considerably worse than yours. You do 0.33 a second.The real reason to prefer it at your size: you can enqueue the job in the same transaction as the row that caused it. Order rolls back, receipt job never existed. With a separate queue you get the classic pair of bugs - charged but no receipt, or receipt sent for an order that rolled back - and the standard fix for those is an outbox table, which is a Postgres queue with extra steps.
Reply
Report
@kombucha_kai · 8mo ago
This is the part that gets skipped in every "which queue" thread. "Redis is faster" is true and irrelevant at 20 a minute. "Redis is not in your transaction" is the thing that costs you a weekend and a customer apology.
Reply
Report
@finops_reyna · 8mo ago
Also one fewer thing in the restore story. Restoring a database backup and having your pending job state come back consistent with it is quietly wonderful the one time you need it.
Reply
Report