Warehouse bill tripled the month we added one incremental model Cost & Scale
One new model, runs hourly, reads from a large event table and writes about 400k rows a day. The bill went from roughly $900 to $2700 a month and this is the only change we made. The model looks cheap when I run it once by hand. Where does this kind of cost usually hide?
@coldstorage_cy · 2mo ago · 3 replies
Almost certainly full table scans on the source, 24 times a day instead of once.
Incremental models are incremental on the write side. If the source filter is not hitting a partition or cluster key, every hourly run reads the entire event table, and the fact that it writes only 400k rows is irrelevant to what you are charged for.
Check in this order:
event_timewhen the table is partitioned onload_datemeans you scan everything.where ts > (select max(ts) from this)? Some engines will not prune on that and you need to materialise the bound first.Run the model once, read the bytes scanned figure, multiply by 720. That number is your answer and it usually explains the whole increase.
Reply
Report
@coldstorage_cy · 2mo ago
Stick it on the wall. Hourly schedules turn a shrug into a mortgage.
Reply
Report
@airgap_amir · 2mo ago
Bytes scanned times 720 is a brutal little formula and I have never once applied it.
Reply
Report