Ask
63
@carryon_only ·

is it normal that touching one shared package rebuilds all 14 apps in ci Turborepo

First time working in a monorepo, six months in and I still cannot tell what is a design decision and what is a mistake nobody has fixed.

We have a @acme/utils package that basically everything depends on. If I change one function in it, CI builds and tests all 14 apps. That is 22 minutes. If I change a file in one app it is 90 seconds.

My instinct says the 22 minutes is correct - I did change something everything depends on. But it also means the smallest possible edit to that package costs me half an hour and people have started avoiding touching it, which seems worse than the build time.

Is this just what monorepos are, or is something set up wrong?

7 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @dartfit_dana · 7mo ago · 2 replies

    Half right. Rebuilding everything downstream is correct and you do not want to change it - that is the whole point of the dependency graph and it is what stops you shipping a broken app because you changed a utility.

    The part that is not normal is that it costs 22 minutes of wall clock every time. Two things fix that without giving up correctness:

    1. Remote cache. If a package's inputs did not change, its build should be a download, not a build. When @acme/utils changes, everything downstream genuinely does need rebuilding, but on a PR that touches one app, the other 13 should cost a few seconds each. If they do not, your caching is broken and that is the actual bug.
    2. Only test what is affected. Building everything downstream is cheap once cached. Running every test suite is not.
    turbo run test --filter=...[origin/main]
    

    runs tests for changed packages and everything that depends on them, and skips the rest. On a utils change that is still most of the repo, but on the 90% of PRs that touch one app it is a fraction.

    The fact that people are avoiding the shared package is the real signal here. That is a build system problem turning into a code quality problem, and it always ends with three slightly-different copies of the same helper.

    58
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @carryon_only · 7mo ago

      We do not have remote caching at all. Everything is built from scratch on every run. Suddenly the 22 minutes makes a lot more sense as a symptom rather than a fact of life.

      20
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @whetstone_wu · 7mo ago · 2 replies

    The other half is that @acme/utils is doing too much. A package that all 14 apps depend on is a package where every change is maximally expensive, so the pressure is to split it along the lines of who actually uses what.

    When we did this - one utils became four smaller packages - the median blast radius of a change dropped from 14 apps to 3, without any build tooling changes at all. Cheapest performance work I have ever done.

    Sign you need this: the package has no coherent one-sentence description. "Utils" is not a description, it is a place things go when nobody decided where they belong.

    41
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @two_calendars · 7mo ago

      Counterpoint from the other side of this: we split too aggressively and ended up with 40 packages where a single feature touches six of them and every PR is a version-bump ceremony. There is a middle. Split when a change to one part keeps forcing rebuilds of consumers who do not care about that part, not on principle.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @two_calendars · 7mo ago

    Worth internalising the distinction: "affected" is about the dependency graph, "changed" is about git. Most tooling gives you both and people conflate them. You want changed for deciding what to cache, affected for deciding what to verify.

    27
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @couchto5kagain · 7mo ago

    The behavioural bit you noticed is the important part of the post. Once a directory becomes expensive to touch, people route around it, and the codebase quietly gets worse in ways no dashboard shows. I would raise it that way with your team rather than as a CI complaint - it lands better.

    15
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @secondshooter_v · 7mo ago

    Six months in and asking whether the thing everyone accepts is actually correct. Keep doing that.

    5
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report