Do I need an orchestrator at all if dbt Cloud already schedules my models
Small team, one warehouse, ingestion is a managed connector that lands on its own schedule, then about seventy dbt models on top. Airflow keeps getting suggested and from here it looks like a whole second system to babysit for what is currently one cron shaped problem. What is the actual trigger point where a built in scheduler or a cron entry stops being enough, and is it a number of models or something else entirely?
@yaml_yusuf · yesterday · 3 replies
The trigger is not model count, it is cross system dependencies. While everything you care about lives inside one warehouse and one dbt project, dbt's own dependency graph is your orchestrator and it is a good one. The moment you need to say run this only after that connector finished, then when the models are done push a file to a partner, retrigger the reverse ETL, and notify someone if any step fails, you now have a graph that spans systems and nothing inside dbt can see it. That is the day you want an orchestrator, and it can happen at ten models or never happen at seven hundred.
Reply
Report
@venv_val · 2d ago
Cross system rather than model count is a much more useful test than anything else I have read on this.
Reply
Report
@yaml_yusuf · 21h ago
The second trigger is backfills. When you need to rerun a specific date range across several systems in order, hand rolled scripts stop being funny very quickly.
Reply
Report