Ask
63
@venv_val ·

Do I need an orchestrator at all if dbt Cloud already schedules my models

Small team, one warehouse, ingestion is a managed connector that lands on its own schedule, then about seventy dbt models on top. Airflow keeps getting suggested and from here it looks like a whole second system to babysit for what is currently one cron shaped problem. What is the actual trigger point where a built in scheduler or a cron entry stops being enough, and is it a number of models or something else entirely?

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @yaml_yusuf · yesterday · 3 replies

    The trigger is not model count, it is cross system dependencies. While everything you care about lives inside one warehouse and one dbt project, dbt's own dependency graph is your orchestrator and it is a good one. The moment you need to say run this only after that connector finished, then when the models are done push a file to a partner, retrigger the reverse ETL, and notify someone if any step fails, you now have a graph that spans systems and nothing inside dbt can see it. That is the day you want an orchestrator, and it can happen at ten models or never happen at seven hundred.

    97
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @venv_val · 2d ago
      @venv_val OP ·

      Cross system rather than model count is a much more useful test than anything else I have read on this.

      28
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @yaml_yusuf · 21h ago

      The second trigger is backfills. When you need to rerun a specific date range across several systems in order, hand rolled scripts stop being funny very quickly.

      33
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @starschema_sy · 2d ago · 2 replies

    If you do decide you need one, look at the asset based tools rather than reaching straight for Airflow out of habit. Dagster and Prefect both let you declare the thing being produced rather than the task that produces it, which lines up much better with how dbt already thinks and makes partial reruns natural instead of clever. Airflow is fine and it is everywhere and hiring is easier for it, but its mental model is tasks on a timetable and you will be translating between that and your dbt graph forever.

    72
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @backfill_bram · 4h ago

      The translation tax is real. Every Airflow plus dbt setup I have seen ends up with either one giant task or a home grown parser that turns the manifest into tasks.

      30
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @drip_dana · 2d ago

    Disagreeing with the wait until it hurts crowd, partly. Adopting an orchestrator is much easier when you have five things to move than when you have fifty and three of them are load bearing. I would not install Airflow today at your size, but I would keep the scheduling logic in one file rather than scattered across the dbt Cloud UI, so that the migration is a port rather than an excavation.

    59
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @uptime_hoarder · 2d ago

    Ran cron plus dbt for four years at roughly your size and never regretted it. One shell script, a lock file so two runs cannot overlap, output piped into a log, and a check that alerts if the run has not written a success marker by a given time. The whole thing was under a hundred lines and I could explain it to a new hire in five minutes. Airflow at that size would have been a bigger system than the thing it was scheduling, and I have watched two teams spend more time on their orchestrator than on their models.

    84
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @gradschool_gary · yesterday

    You already have an orchestrator, it is just implicit and made of assumptions: the connector finishes before the schedule fires, usually, and nothing else depends on the output, currently. That works right up until the connector is slow one morning and your models run on yesterday's data without telling anyone. Before adopting anything, add the cheap version of the check, which is a freshness assertion at the top of the run that fails loudly if the source has not landed.

    68
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report