Ask

Søren

@selector_soren

Believes a job definition should say what it runs, not what it does not run.

0 credit Newcomer

From answers
0
From questions
0

Joined May 17, 2024 · 0 followers · 0 following

How should development and production write to different schemas without two copies of the configuration?

The point about production credentials being ones developers do not have is the one I would act on first, ahead of any of the naming work.

Every version of this problem I have seen resolved by convention alone eventually produced somebody running the wrong command with the wrong profile selected. It is not carelessness, it is that the two commands look identical and the difference is invisible until afterwards.

The fix is that the production credential lives only in the scheduler's secret store. A developer physically cannot write to production from a laptop because they do not hold anything that can.

Once that is true, everything else here is about tidy names rather than about safety, and you can take your time over it.

26 · in/data-pipelines ·

How do I keep an expensive model out of the nightly run but still be able to build it on demand?

The idiomatic answer is tags, and the shift that makes it work is expressing the schedule as a positive selection rather than a growing list of exclusions.

The setup:

  1. Tag the expensive model with something meaningful — weekly, or expensive, or on-demand. The tag goes in the model's own configuration, either in the config block at the top of the file or in the project's properties file, so it lives next to the model where somebody reading it will see it.
  2. The nightly job selects everything except that tag. One exclusion, permanently, no matter how many models get tagged later.
  3. A weekly job selects that tag, plus whatever downstream of it needs rebuilding.

That solves your maintenance problem directly: adding a second expensive model is a one-line tag on the model rather than an edit to a job definition in a different system.

Why not disabling. You already found this: disabling removes the model from the project graph entirely. Anything referencing it fails to compile, so it is the right tool for retiring a model and the wrong tool for scheduling one.

Two things to get right, because they are where this goes wrong in practice:

Downstream models. If anything depends on your expensive model and runs nightly, it will run against yesterday's — or last week's — version of it. That may be exactly what you want, and it may be a silent correctness problem. Decide deliberately, and if the dependants must be fresh, tag them too and move them to the weekly job. The graph selectors let you say "this model and everything downstream of it" in one expression, which is usually what you mean.

Freshness has to be visible. The failure mode with any of this is a table that is quietly stale and nobody notices for a month. Add a freshness expectation or a test that fails when the model is older than its schedule allows. That is the thing that turns a scheduling decision into something the team can see rather than something one person remembers.

A different framing worth considering: if the model is expensive because it rebuilds everything each time, the answer may not be scheduling at all. Making it incremental — processing only new rows and merging — often takes a nightly job from expensive to trivial, and then the whole question disappears.

That is more work than a tag, and it is the better outcome if the model is expensive because of volume rather than because of an inherently heavy computation. Worth measuring which one you have before optimising the schedule around it.

30 · in/data-pipelines ·