145
@airgap_amir ·

Warehouse bill tripled the month we added one incremental model Cost & Scale

One new model, runs hourly, reads from a large event table and writes about 400k rows a day. The bill went from roughly $900 to $2700 a month and this is the only change we made. The model looks cheap when I run it once by hand. Where does this kind of cost usually hide?

7 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @coldstorage_cy · 2mo ago · 3 replies

    Almost certainly full table scans on the source, 24 times a day instead of once.

    Incremental models are incremental on the write side. If the source filter is not hitting a partition or cluster key, every hourly run reads the entire event table, and the fact that it writes only 400k rows is irrelevant to what you are charged for.

    Check in this order:

    • Is the source partitioned on the column your incremental filter uses? Filtering on event_time when the table is partitioned on load_date means you scan everything.
    • Does your filter wrap the column in a cast or a date function? That frequently defeats partition pruning.
    • Is the predicate a subquery like where ts > (select max(ts) from this)? Some engines will not prune on that and you need to materialise the bound first.

    Run the model once, read the bytes scanned figure, multiply by 720. That number is your answer and it usually explains the whole increase.

    138
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @coldstorage_cy · 2mo ago

      Stick it on the wall. Hourly schedules turn a shrug into a mortgage.

      46
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @airgap_amir · 2mo ago

      Bytes scanned times 720 is a brutal little formula and I have never once applied it.

      40
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @dbt_and_dust · 2mo ago · 2 replies

    Also question the cadence. Ask who consumes it and how fresh they actually need it. Half the hourly models I inherit are hourly because somebody typed hourly, and moving to every four hours cuts the bill by 75 percent with nobody noticing.

    88
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @proof_pilar · 2mo ago

      Ask them what decision they make with it. If nobody can name one, it can run daily.

      34
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @sheetsmith · 2mo ago

    Check for a full refresh happening on every run. One misconfigured flag and your 'incremental' model rebuilds from scratch hourly, which fits your numbers exactly.

    55
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @goroutine_gil · 2mo ago

    Tag models and set a cost alert if your platform allows it. Finding this on a bill a month later is the expensive way to learn.

    25
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report