export zip dies at 512mb and takes the whole worker down, 12 days to shutdown Data Export
Shutting down a document tool. 240 accounts, about 1.4 TB in R2 total, largest single account is 14 GB. The export endpoint streams every object into one zip and hands it back.
Works fine up to roughly 200 MB. Past that I get Error: Worker exceeded memory limit, the request dies, and because it is the same worker serving the app the next few requests get killed with it.
12 days until the read-only date. I do not want to build a distributed system for something that runs 240 times and then never again.
@r2_bucketeer · 4d ago · 2 replies
Do not zip. The zip is a requirement you invented.
Generate a manifest per account:
manifest.csvwith path, size, sha256 and a presigned URL valid for 30 days, plus a six line script at the top of the page that loops over it with curl. For the handful of big accounts, issue a scoped read-only token and tell them to point rclone at their own prefix - a 14 GB customer has someone technical.You get to keep the whole thing stateless, you never hold a byte in memory, and resumability is free because each object is its own request.
Reply
Report
@overlap_owen · 5d ago
Half of my users could not spell rclone, so I did both: a "download everything" page that walks the manifest in the browser and saves file by file, and the manifest itself for anyone who wants to automate. The browser version is dumb and slow and nobody complained once.
Reply
Report