Ask
27

How do I keep an expensive model out of the nightly run but still be able to build it on demand?

One of our models is expensive and only needs rebuilding weekly, or when somebody asks for it. Right now it runs every night with everything else because the scheduled job builds the whole project.

Disabling it in configuration removes it from the project entirely, which is not what I want: I still need to be able to build it deliberately, and I still want its downstream dependencies to resolve.

Excluding it by name in the nightly job works, but the exclusion list is growing and it lives in the job configuration rather than next to the model, so nobody discovers it until they wonder why something is stale.

What is the idiomatic way to express this?

10 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @selector_soren · 4w ago · 3 replies

    The idiomatic answer is tags, and the shift that makes it work is expressing the schedule as a positive selection rather than a growing list of exclusions.

    The setup:

    1. Tag the expensive model with something meaningful, weekly, or expensive, or on-demand. The tag goes in the model's own configuration, either in the config block at the top of the file or in the project's properties file, so it lives next to the model where somebody reading it will see it.
    2. The nightly job selects everything except that tag. One exclusion, permanently, no matter how many models get tagged later.
    3. A weekly job selects that tag, plus whatever downstream of it needs rebuilding.

    That solves your maintenance problem directly: adding a second expensive model is a one-line tag on the model rather than an edit to a job definition in a different system.

    Why not disabling. You already found this: disabling removes the model from the project graph entirely. Anything referencing it fails to compile, so it is the right tool for retiring a model and the wrong tool for scheduling one.

    Two things to get right, because they are where this goes wrong in practice:

    Downstream models. If anything depends on your expensive model and runs nightly, it will run against yesterday's - or last week's: version of it. That may be exactly what you want, and it may be a silent correctness problem. Decide deliberately, and if the dependants must be fresh, tag them too and move them to the weekly job. The graph selectors let you say "this model and everything downstream of it" in one expression, which is usually what you mean.

    Freshness has to be visible. The failure mode with any of this is a table that is quietly stale and nobody notices for a month. Add a freshness expectation or a test that fails when the model is older than its schedule allows. That is the thing that turns a scheduling decision into something the team can see rather than something one person remembers.

    A different framing worth considering: if the model is expensive because it rebuilds everything each time, the answer may not be scheduling at all. Making it incremental - processing only new rows and merging - often takes a nightly job from expensive to trivial, and then the whole question disappears.

    That is more work than a tag, and it is the better outcome if the model is expensive because of volume rather than because of an inherently heavy computation. Worth measuring which one you have before optimising the schedule around it.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @tag_sprawl · 3w ago · 2 replies

      Worth a warning about where tags go if nobody owns them. We started with two and now have nineteen, several of which overlap, and the nightly job selects on a union of four of them that nobody can explain.

      Tags solve the exclusion list problem by turning it into a naming problem, which is better but not free. If you add one, write down what it means the same day.

      22
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @selector_soren · 3w ago

        Fair, and it is the same failure as the exclusion list, one level up. The answer is the same too: the selection lives in the project, not in the job.

        14
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report
  • @expensive_model_ed · 4w ago · 2 replies

    On making it incremental: check what the model is actually spending its time on before assuming volume is the cause.

    I had a model I was sure was expensive because of size. It was expensive because of one join with a badly-chosen key that was fanning out and then being deduplicated at the end. Fixing the join took the run from twenty minutes to forty seconds and I never needed the schedule change at all.

    The warehouse's query profile tells you this in a couple of minutes. Worth looking before designing around the cost.

    If it genuinely is volume, incremental is the answer, and the thing to be careful about is late-arriving data - an incremental model that only looks at the newest rows will miss records that arrive with an older timestamp. Give yourself an overlap window.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @cost_per_model · 3w ago

      Adding a number to the point about checking what it actually spends time on: put the cost per run somewhere visible. Ours went on a dashboard and two of the three expensive models turned out to be expensive for reasons we could fix in an afternoon, which is a much better outcome than scheduling them less often.

      18
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @nightly_at_0200 · 2w ago · 2 replies

    Is there a reason not to just give the weekly model its own job rather than tagging it out of the nightly one

    11
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @selector_soren · 2w ago

      You can, and it is fine at one model. It stops working at five, because now the schedule lives in however many job definitions and you cannot answer "what runs tonight" without reading all of them. The tag version keeps that answer in one place.

      16
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @nightly_at_0200 · 3w ago

    The exclusion list growing in the job config is such a recognisable smell. Ours had a comment on it saying do not add to this.

    8
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @schema_sibel · 4w ago

    One practical addition: document the tag's meaning in the project, not just in someone's head.

    A tag called weekly is only self-explanatory until there are three of them and one of them means "run weekly" while another means "do not run automatically at all". Write the convention down in the project's readme with one line per tag, and the next person can read the tag on a model and know what it implies.

    The cost of not doing this is a model tagged for a reason nobody remembers, which nobody dares change.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @fuse_mount_fahri · 4w ago

    Worth making the on-demand run genuinely easy to trigger, or people stop doing it.

    If building the weekly model requires someone to have the project set up locally with the right credentials, it will happen when that one person is available and not otherwise. A button in the scheduler, or a job that anyone on the team can trigger, changes it from a favour into an operation.

    Sounds like a small thing. It is the difference between the schedule you designed and the schedule you actually get.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report