Built our first real scheduled pipeline about six weeks ago. It has needed manual intervention four times: an API rate limit, an upstream schema change, a disk filling up, and one thing I still cannot explain. I expected to build it and mostly forget it. Is four times in six weeks a sign I built it badly, or is this the job?
Add a contract test on the upstream schema. Ten lines that fail loudly when a column type changes turn a mysterious 3am mess into a clear message before anything downstream runs.
Do you fail the run on a schema change, or warn and carry on with the columns you know?
Fail on a type change or a missing column, warn on a new column. New columns are usually harmless and blocking on them trains people to ignore the alert.
The first six weeks is when you find all the assumptions. It gets quieter. It never becomes zero.
Disk filling up is the one that is genuinely on you, and it will happen again. Set the alert at 70 percent, not 90.
We had no alert at all, which is how it became an incident rather than a chore.
Add a retention rule on the staging area at the same time. Almost every disk-full I have seen was old intermediate output nobody meant to keep.
@coldstorage_cy · 10mo ago · 3 replies
Four in six weeks for a first pipeline is unremarkable, and three of your four causes were external. Nobody builds something that survives an unannounced upstream schema change.
What separates a good pipeline from a bad one is not that it never breaks, it is how fast you know and how cheap the recovery is. Ask yourself:
If the answers are monitor, rerun and yes, you are doing this correctly and the count does not matter. The unexplained one is the only item on your list I would chase. Unexplained failures come back.
Reply
Report
@coldstorage_cy · 10mo ago
That is the one to fix. Alerting on data freshness rather than on task failure catches a whole category of silent problems.
Reply
Report
@proof_pilar · 10mo ago
Two out of three. We found out from a person the first time and that was easily the worst one.
Reply
Report