How do I keep an expensive model out of the nightly run but still be able to build it on demand?
One of our models is expensive and only needs rebuilding weekly, or when somebody asks for it. Right now it runs every night with everything else because the scheduled job builds the whole project.
Disabling it in configuration removes it from the project entirely, which is not what I want — I still need to be able to build it deliberately, and I still want its downstream dependencies to resolve.
Excluding it by name in the nightly job works, but the exclusion list is growing and it lives in the job configuration rather than next to the model, so nobody discovers it until they wonder why something is stale.
What is the idiomatic way to express this?
@selector_soren · 15h ago
The idiomatic answer is tags, and the shift that makes it work is expressing the schedule as a positive selection rather than a growing list of exclusions.
The setup:
weekly, orexpensive, oron-demand. The tag goes in the model's own configuration, either in the config block at the top of the file or in the project's properties file, so it lives next to the model where somebody reading it will see it.That solves your maintenance problem directly: adding a second expensive model is a one-line tag on the model rather than an edit to a job definition in a different system.
Why not disabling. You already found this: disabling removes the model from the project graph entirely. Anything referencing it fails to compile, so it is the right tool for retiring a model and the wrong tool for scheduling one.
Two things to get right, because they are where this goes wrong in practice:
Downstream models. If anything depends on your expensive model and runs nightly, it will run against yesterday's — or last week's — version of it. That may be exactly what you want, and it may be a silent correctness problem. Decide deliberately, and if the dependants must be fresh, tag them too and move them to the weekly job. The graph selectors let you say "this model and everything downstream of it" in one expression, which is usually what you mean.
Freshness has to be visible. The failure mode with any of this is a table that is quietly stale and nobody notices for a month. Add a freshness expectation or a test that fails when the model is older than its schedule allows. That is the thing that turns a scheduling decision into something the team can see rather than something one person remembers.
A different framing worth considering: if the model is expensive because it rebuilds everything each time, the answer may not be scheduling at all. Making it incremental — processing only new rows and merging — often takes a nightly job from expensive to trivial, and then the whole question disappears.
That is more work than a tag, and it is the better outcome if the model is expensive because of volume rather than because of an inherently heavy computation. Worth measuring which one you have before optimising the schedule around it.
Reply
Report