Cy

So the day four panic sniff is basically the least informative test available.

I will time it properly next batch. I was definitely eyeballing what felt like two minutes.

Almost certainly full table scans on the source, 24 times a day instead of once.

Incremental models are incremental on the write side. If the source filter is not hitting a partition or cluster key, every hourly run reads the entire event table, and the fact that it writes only 400k rows is irrelevant to what you are charged for.

Check in this order:

• Is the source partitioned on the column your incremental filter uses? Filtering on event_time when the table is partitioned on load_date means you scan everything.
• Does your filter wrap the column in a cast or a date function? That frequently defeats partition pruning.
• Is the predicate a subquery like where ts > (select max(ts) from this)? Some engines will not prune on that and you need to materialise the bound first.

Run the model once, read the bytes scanned figure, multiply by 720. That number is your answer and it usually explains the whole increase.

Stick it on the wall. Hourly schedules turn a shrug into a mortgage.

The run_id in the staging path is the piece we are missing. We overwrite the same staging prefix every night, which is exactly why a partial write is poison.

We merge on an API id that I am now realising nobody has ever verified.

The middle ground is a rolling reprocess window. You know your late arrival horizon is about four days, so rebuild the last five or seven days every night instead of only yesterday. That costs roughly seven times one day, which is nothing next to a 90 day backfill, and your numbers become correct within the window automatically.

Two details that matter:

• Partition by event date, not ingestion date, and always reprocess whole partitions.
• Keep the ingestion timestamp as a column anyway. It lets you answer 'what did this number look like on Tuesday' when finance asks, and they will.

If you have a genuine long tail beyond four days, add a weekend job that reprocesses the last 45 days. Nobody notices a correction that lands on a Sunday.

Because the tutorial always shows yesterday. Almost every real pipeline has a window.

Four in six weeks for a first pipeline is unremarkable, and three of your four causes were external. Nobody builds something that survives an unannounced upstream schema change.

What separates a good pipeline from a bad one is not that it never breaks, it is how fast you know and how cheap the recovery is. Ask yourself:

• Did you find out from a monitor, or from a person asking why the dashboard was empty?
• Could you rerun the failed day safely, or did it need surgery?
• Did you add a check after each incident so that class of failure is now noisy rather than silent?

If the answers are monitor, rerun and yes, you are doing this correctly and the count does not matter. The unexplained one is the only item on your list I would chase. Unexplained failures come back.

That is the one to fix. Alerting on data freshness rather than on task failure catches a whole category of silent problems.

That minute is the most expensive minute in your day and it teaches you nothing.

Do both for a fortnight. Build the merged version into a shadow table, compare row counts and a few aggregates against the rebuild every day, and you find out whether your keys hold up without betting anything on it.