Ed

On making it incremental — check what the model is actually spending its time on before assuming volume is the cause.

I had a model I was sure was expensive because of size. It was expensive because of one join with a badly-chosen key that was fanning out and then being deduplicated at the end. Fixing the join took the run from twenty minutes to forty seconds and I never needed the schedule change at all.

The warehouse's query profile tells you this in a couple of minutes. Worth looking before designing around the cost.

If it genuinely is volume, incremental is the answer, and the thing to be careful about is late-arriving data — an incremental model that only looks at the newest rows will miss records that arrive with an older timestamp. Give yourself an overlap window.

If it does turn out to be a daily cap, consider whether the job needs to move that much data at all.

A sync that reuploads files it has already sent — because a timestamp changed, or a checksum is not being compared, or something is rewriting files in place — burns the allowance for no benefit. Look at what was actually transferred in the last few successful runs; if the volume is roughly constant and roughly the size of the whole dataset, it is not doing an incremental sync and that is the real problem.

Worth adding: whatever convention you land on, make the target visible in the run output and in anything that reports on the project.

The worst version of this is somebody spending an afternoon confused about why their changes are not showing up, because they were building into one schema and querying another. It is invisible unless something tells you.

Most tooling will print the target at the start of a run. Get people in the habit of reading that line, and put the target in the name of any dashboard or notebook that reads from these tables.