Ask
96
@durable_ines ·

r2 bill hit $214 from class a operations because i list the bucket per request Bill Breakdown

R2 was supposed to be the cheap part. 90GB stored, zero egress charges, I was budgeting maybe $2 a month for it.

Invoice came in at $214 and almost all of it is Class A operations. Working backwards from the per-million price that is roughly 47 million operations in a month.

Found it: my gallery page calls ListObjectsV2 on the bucket prefix to build the file list, on every single page view. About 1.6M page views a month, and the listing is paginated so heavy folders are several calls each. So every visitor is doing bucket listings that could have been one database query.

So it is my fault and I know the fix in the abstract - do not use object storage as a database - but I would like to hear how people actually structure this, because "the bucket is the source of truth" felt clean and now it costs more than my server.

10 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @kraut_corner · last wk. · 2 replies

    Short-term tourniquet before you rewrite anything: cache the listing.

    const key = new Request(`https://cache.local/list/${prefix}`)
    let res = await caches.default.match(key)
    if (!res) {
      res = new Response(JSON.stringify(await listPrefix(prefix)), {
        headers: { 'cache-control': 'max-age=60' }
      })
      ctx.waitUntil(caches.default.put(key, res.clone()))
    }
    

    Sixty seconds of staleness on a gallery is nothing and it removes about 99% of the operations immediately. You can then do the database version properly instead of at 2am.

    Be aware the edge cache is per-location, so a 60s TTL across many colos is not one call a minute globally, it is one call a minute per colo that gets traffic. Still a ~99% cut, just not 100%.

    54
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @dartfit_dana · last wk.

      Worth stating explicitly for anyone skimming: this is a patch, not the fix. Caching a bad access pattern makes the bill survivable and leaves the design in place. Do the tourniquet today and the DB index this week, not this quarter.

      22
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @egress_egon · last wk. · 3 replies

    The mental model that fixes this permanently: object storage is a place to put bytes, not a place to ask questions. Any question - what exists, what changed, what belongs to this user - is a database query.

    What that looks like concretely:

    • one row per object in D1 (or Postgres, whatever you have): key, size, content type, owner, created_at, plus whatever you want to filter on.
    • writes go through your code, so you insert the row in the same request that does the PUT.
    • the gallery reads the DB. R2 is only ever touched to serve or store the actual bytes.
    • a nightly reconciliation job lists the bucket once and fixes drift, because drift will happen.

    That nightly job is one Class A operation per 1,000 objects, once a day. Your 47 million becomes something like 3,000 a month.

    The pricing asymmetry is the thing to burn into memory: Class A (writes and lists) costs more than ten times what Class B (reads) costs. Any design where a read path triggers a Class A op is going to be expensive, and it will be expensive in proportion to your traffic, which is exactly the wrong direction.

    79
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @durable_ines · last wk.

      The reconciliation job is the piece I was missing. My objection to a DB index was always "what if they diverge" and the answer is apparently "they will, check nightly, move on".

      23
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @kraut_corner · last wk.

      Divergence is also mostly one-directional in practice - orphaned objects in the bucket with no row, because a request died between PUT and INSERT. Those are harmless until they cost storage. Rows with no object are the scary case and you can make that nearly impossible by writing the row after the PUT succeeds.

      19
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @dartfit_dana · 7d ago

    While you are auditing, remember PUT, COPY and POST are also Class A. The other common way people generate a surprise bill is an idempotent-looking sync that re-uploads unchanged files because it compares timestamps instead of content hashes. Every no-op sync is a full-price write.

    If you have anything on a schedule that "makes sure the bucket matches", go read it now.

    33
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @pgpolicy_nadia · last wk.

    If you go the D1 route, put an index on (owner_id, created_at desc) from the start and store the key rather than a full URL. Signed URLs expire, keys do not, and you will want to change your delivery layer at some point without rewriting every row.

    18
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @gpio_gwen · last wk. · 2 replies

    I thought R2's whole pitch was no request charges - the entire point is that you only pay for storage and the egress is free. Are you sure this is not something else on the invoice mislabelled?

    9
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @egress_egon · last wk.

      No - the pitch is no egress charges. Operations are billed in two classes, and Class A is the expensive one at several dollars per million. It is genuinely cheap for normal usage, which is why nobody thinks about it until they put a list call in a hot path and do 47 million of them.

      7
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @kraut_corner · last wk.

    "the bucket is the source of truth" is one of those sentences that is architecturally beautiful and financially expensive.

    4
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report