Events can arrive up to four days after the event timestamp. My daily rollups are built once at 2am for the previous day, so anything late is simply missing until somebody notices and asks for a backfill, which reprocesses 90 days and takes about six hours. There must be a middle ground between rebuilding everything and being wrong.
Measure the tail before you pick a window. Plot ingestion time minus event time over a couple of months. People usually find 99.5 percent lands within a day and there is a thin weird tail from one broken client that should be fixed at the source instead.
And that thin tail is very often a single mobile app version with a bad offline queue.
Six hours for 90 days suggests the backfill is not parallel across partitions. If each day is independent they should run concurrently, and at your volume that number should be closer to twenty minutes.
They are sequential. That is embarrassing and also very fixable.
Publish a 'data complete through' timestamp next to the tables so consumers know yesterday is provisional. Half the pain here is people trusting a number that was never final.
@coldstorage_cy · 8mo ago · 3 replies
The middle ground is a rolling reprocess window. You know your late arrival horizon is about four days, so rebuild the last five or seven days every night instead of only yesterday. That costs roughly seven times one day, which is nothing next to a 90 day backfill, and your numbers become correct within the window automatically.
Two details that matter:
If you have a genuine long tail beyond four days, add a weekend job that reprocesses the last 45 days. Nobody notices a correction that lands on a Sunday.
Reply
Report
@coldstorage_cy · 8mo ago
Because the tutorial always shows yesterday. Almost every real pipeline has a window.
Reply
Report
@dbt_and_dust · 8mo ago
Rolling seven days nightly plus a weekend sweep is nearly free for us. I do not know why I was thinking in terms of one day or all of it.
Reply
Report