Ask
141

export zip dies at 512mb and takes the whole worker down, 12 days to shutdown Data Export

Shutting down a document tool. 240 accounts, about 1.4 TB in R2 total, largest single account is 14 GB. The export endpoint streams every object into one zip and hands it back.

Works fine up to roughly 200 MB. Past that I get Error: Worker exceeded memory limit, the request dies, and because it is the same worker serving the app the next few requests get killed with it.

12 days until the read-only date. I do not want to build a distributed system for something that runs 240 times and then never again.

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @r2_bucketeer · 4d ago · 2 replies

    Do not zip. The zip is a requirement you invented.

    Generate a manifest per account: manifest.csv with path, size, sha256 and a presigned URL valid for 30 days, plus a six line script at the top of the page that loops over it with curl. For the handful of big accounts, issue a scoped read-only token and tell them to point rclone at their own prefix - a 14 GB customer has someone technical.

    You get to keep the whole thing stateless, you never hold a byte in memory, and resumability is free because each object is its own request.

    165
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @overlap_owen · 5d ago

      Half of my users could not spell rclone, so I did both: a "download everything" page that walks the manifest in the browser and saves file by file, and the manifest itself for anyone who wants to automate. The browser version is dumb and slow and nobody complained once.

      41
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @merino_max · 5d ago · 2 replies

    If you insist on a zip, stream it and turn compression off. That runtime gives you a fixed 128 MB heap, and a zip writer that buffers entries or builds the central directory in memory will die exactly where yours is dying. A streaming zip over a TransformStream with store-only entries never holds more than a chunk.

    Store-only is also the correct call on the merits - your uploads are already compressed formats, so you are burning CPU to turn 14 GB into 13.8 GB.

    94
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @nadia_brill · 3d ago

      Store-only has the nice side effect that you can compute the exact final size in advance from the object sizes, so you can show a real progress bar and set Content-Length instead of chunking blind.

      25
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @thriftedtweed · 5d ago

    Run it offline, once per account, and stop treating it as an endpoint. 240 runs is a laptop, a for loop and a weekend. Write each zip straight into R2 with multipart upload, then email a link. No memory limit, no request timeout, no user waiting on a spinner, and if run 137 fails you rerun run 137.

    52
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @shopvac_ghost · 3d ago · 2 replies

    Bump the memory in wrangler.toml, there is a limits block for exactly this - set memory_mb higher and the problem goes away without rewriting anything.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @lunchboxrotation · 3d ago

      The limits block covers CPU time, not heap. Worker memory is fixed and is not something you can buy more of on that runtime. If you genuinely need memory you need different compute - a container, a VM, anything with a knob. Sending OP off to edit a config that has no effect is a bad two hours out of the twelve days they have left.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @adjunct_life · 2d ago

    Whatever you ship, count exports per account and put the number on a page you look at every morning. On mine, 31 of 84 accounts never downloaded anything, and those are precisely the people who email you in March asking whether there is any way to get their data back.

    11
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report