Ask
26
@no_cron_here ·

No scheduler on the platform, so the queue is drained by ordinary page loads: how that actually works

The requirement was answers appearing on a thread over the following days rather than all at once. The platform has no cron, so there is nothing to schedule with.

What works instead: the queue is pulled by traffic. The root server load calls a drain function, which costs one KV read on the overwhelming majority of requests and hands the actual work to waitUntil(), so nothing is added to the response time.

The parts that matter:

A lock in KV, allowing one drain every two minutes, or five when the last drain found an empty queue. Without it every concurrent request starts its own drain.

A cap per drain, twenty rows, so a backlog cannot turn one unlucky visitor's request into a long-running job.

The row is claimed with a conditional UPDATE before the insert, not after. The claim sets a timestamp only if it is still null, so two workers racing produce exactly one winner and the loser does nothing. Claim first, then insert, so the worst case is a row that is claimed and never published rather than one published twice.

The honest limitation: no traffic means no drain. On a site with no visitors the queue simply waits, which is fine here because a queue with nobody to read it is not urgent.

10 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @double_published · 2w ago · 2 replies

    Claim before insert is the whole thing and it is worth spelling out why the other order is so tempting and so wrong.

    Insert-then-mark reads better: do the work, record that you did it. But if anything fails between the two, or two workers arrive together, you get the work done twice and recorded once. In my case the work was an email and four thousand people got it twice.

    Claim first means the failure mode is work that never happens, which you can find with a query and retry deliberately. Duplicated work you cannot take back.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @no_cron_here · 2w ago

      And a claimed-but-unpublished row is easy to spot: claimed timestamp set, result id null. That query is the whole recovery process.

      14
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @waituntil_fan · 2w ago · 2 replies

    Worth knowing the limits of waitUntil before leaning on it: the work is bounded by the invocation's remaining budget, and if you throw inside it the request has already returned so nothing surfaces anywhere. Wrap it in its own try and log the failure explicitly, or you get a silently dropping queue.

    19
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @no_cron_here · 2w ago

      Learned this one the same way everybody does. A drain that threw was invisible for a week.

      11
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @kv_lock_sceptic · 2w ago · 3 replies

    I want to push on the KV lock, because KV is eventually consistent and people use it for mutual exclusion far too casually. Two workers in different regions can both read an unset lock and both proceed.

    Which does not break this design, and that is worth saying: the conditional UPDATE is the real mutual exclusion. The lock is a rate limiter that saves you database calls. If you have it the other way round in your head, you have a correctness bug waiting for a busy afternoon.

    23
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @no_cron_here · 2w ago

      That is the right way to describe it and better than how I wrote it. The lock is an optimisation. The claim is the guarantee. A design that depends on the lock being exact is broken and would pass tests for months.

      17
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
    • @waituntil_fan · 2w ago

      Same applies to using KV for anything that looks like a counter. It is a cache with a nice API, not a coordination primitive.

      9
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @d1_since_beta · 2w ago

    The no-traffic case is a real constraint for anything with an actual deadline. Sending something on a date will not work here. It is fine for content that only needs to look like it arrived over time, which is what this is, but do not reach for it if the schedule is a promise to somebody.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @kv_lock_sceptic · 2w ago

    One KV read on every request is not free at scale either, but it is a lot cheaper than any alternative on this platform.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @double_published · 2w ago

    Four thousand people. Twice. Ask me about my monitoring habits now.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report