crawler on a worker saves the rendered html for each page it visits, roughly 40k snapshots a month, and each one is 1 to 3 mb of text. i need to query by url and date and occasionally diff two versions. putting the blob straight in a d1 column is one less moving part, so before i build the two-store version i'd like to know whether anyone regrets doing it either way.
Small disagreement with "never put blobs in D1" as a general rule, since people will read this thread later. Under a couple of hundred kilobytes, in a table you rarely select from, it's fine and one store is genuinely simpler.
What kills you is selecting the blob column by accident. D1 responses have to be serialised out of the database and back through the worker, and a SELECT * on a table with fat rows will feel like the network is broken. If you do keep blobs in D1, put them in their own table with the key, so a careless select can't reach them.
Agreed as a general rule, and it doesn't rescue this case — his rows are over the hard limit. But yes, the separate-table trick is the right hedge for the small-blob version.
the SELECT * warning is well timed, my current schema has the blob column in the same table as the crawl metadata i list on a dashboard.
The single-store version isn't a tradeoff, it's not available. D1's maximum size for a string, blob or table row is 2,000,000 bytes — your 3mb snapshots literally cannot be inserted.
And the ones that do fit will end the experiment on storage anyway: a database on the paid plan tops out at 10gb and that ceiling can't be raised. 40k snapshots a month at even 1.5mb average is 60gb a month. You'd hit the wall in about five days.
R2 for the bytes, D1 for url, timestamp, content hash and the object key. That's the design, and it's the design regardless of taste.
Did this wrong on a smaller project: json documents up to about 800kb in a D1 column, no separate table. It worked for two months and then the dashboard query that listed 50 recent items started timing out, because it was pulling 40mb through the worker to render a list of titles.
The fix took an afternoon and I felt stupid the whole time. Metadata and payload never live in the same row unless you always want both.
For the diffing part: store a content hash and the previous object key on each row. Then a diff is two R2 gets and no scanning, and the common case — nothing changed — is answered from D1 alone without fetching anything.
Bytes in R2, facts in D1, hash in both. Compress before writing. There isn't a second good answer at this size.
@leaky_bucket_ed · 2w ago · 3 replies
Numbers for the R2 side so you can sanity check the bill. The free allowance is 10gb-month of standard storage, 1 million class A operations and 10 million class B, and egress is free.
Your writes are class A, so 40k a month is nowhere near the million. Storage is the line that matters — 40k objects at 2mb is roughly 80gb of new data a month, growing. Gzip the html before you store it and text like that typically compresses hard, which changes the storage bill by a lot more than anything else you can do here.
Also store the content hash in D1. Most crawls of the same url return byte-identical html, and skipping the write when the hash matches cut my object count by about 70%.
Reply
Report
@r2_bucketeer · 2w ago
the content hash idea alone probably halves this. most of the pages i'm crawling change once a week at best and i've been storing an identical copy every single run.
Reply
Report
@airgap_amir · 2w ago
Store the hash on the metadata row and you also get change detection for free, which sounds like the diffing feature you were planning to build separately.
Reply
Report