Ask
27

How do I keep an expensive model out of the nightly run but still be able to build it on demand?

One of our models is expensive and only needs rebuilding weekly, or when somebody asks for it. Right now it runs every night with everything else because the scheduled job builds the whole project.

Disabling it in configuration removes it from the project entirely, which is not what I want — I still need to be able to build it deliberately, and I still want its downstream dependencies to resolve.

Excluding it by name in the nightly job works, but the exclusion list is growing and it lives in the job configuration rather than next to the model, so nobody discovers it until they wonder why something is stale.

What is the idiomatic way to express this?

2 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @selector_soren · 15h ago

    The idiomatic answer is tags, and the shift that makes it work is expressing the schedule as a positive selection rather than a growing list of exclusions.

    The setup:

    1. Tag the expensive model with something meaningful — weekly, or expensive, or on-demand. The tag goes in the model's own configuration, either in the config block at the top of the file or in the project's properties file, so it lives next to the model where somebody reading it will see it.
    2. The nightly job selects everything except that tag. One exclusion, permanently, no matter how many models get tagged later.
    3. A weekly job selects that tag, plus whatever downstream of it needs rebuilding.

    That solves your maintenance problem directly: adding a second expensive model is a one-line tag on the model rather than an edit to a job definition in a different system.

    Why not disabling. You already found this: disabling removes the model from the project graph entirely. Anything referencing it fails to compile, so it is the right tool for retiring a model and the wrong tool for scheduling one.

    Two things to get right, because they are where this goes wrong in practice:

    Downstream models. If anything depends on your expensive model and runs nightly, it will run against yesterday's — or last week's — version of it. That may be exactly what you want, and it may be a silent correctness problem. Decide deliberately, and if the dependants must be fresh, tag them too and move them to the weekly job. The graph selectors let you say "this model and everything downstream of it" in one expression, which is usually what you mean.

    Freshness has to be visible. The failure mode with any of this is a table that is quietly stale and nobody notices for a month. Add a freshness expectation or a test that fails when the model is older than its schedule allows. That is the thing that turns a scheduling decision into something the team can see rather than something one person remembers.

    A different framing worth considering: if the model is expensive because it rebuilds everything each time, the answer may not be scheduling at all. Making it incremental — processing only new rows and merging — often takes a nightly job from expensive to trivial, and then the whole question disappears.

    That is more work than a tag, and it is the better outcome if the model is expensive because of volume rather than because of an inherently heavy computation. Worth measuring which one you have before optimising the schedule around it.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @expensive_model_ed · 6h ago

    On making it incremental — check what the model is actually spending its time on before assuming volume is the cause.

    I had a model I was sure was expensive because of size. It was expensive because of one join with a badly-chosen key that was fanning out and then being deduplicated at the end. Fixing the join took the run from twenty minutes to forty seconds and I never needed the schedule change at all.

    The warehouse's query profile tells you this in a couple of minutes. Worth looking before designing around the cost.

    If it genuinely is volume, incremental is the answer, and the thing to be careful about is late-arriving data — an incremental model that only looks at the newest rows will miss records that arrive with an older timestamp. Give yourself an overlap window.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report