Ask
133
@packlist_pen ·

turborepo 2.x remote cache misses on every ci run but hits locally Turborepo

14 packages, pnpm workspaces, turbo run build with remote caching enabled. On my machine the second run is 100% cache hit in about 400ms. On CI it is 0/14 every single time, 6m41s, on a commit that changed one README.

Things I have verified:

  • TURBO_TOKEN and TURBO_TEAM are set on CI, and the run summary does say it is uploading artifacts afterwards. So it can talk to the cache, it just never finds anything.
  • same turbo version locally and on CI (checked with turbo --version).
  • same Node major.
  • --dry=json on CI shows a different globalHash on two consecutive runs of the same commit.

That last one seems like the actual clue - if the global hash changes between two identical runs then obviously nothing can ever hit. I just cannot work out what is feeding into it that is different each time.

What is in the global hash that I am not thinking about?

10 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @dartfit_dana · 3d ago · 3 replies

    Two identical runs producing different global hashes means an environment variable is in the hash and its value changes per run. That is basically the only thing it can be.

    Dump the inputs and read them:

    turbo run build --dry=json > dry.json
    jq '.globalCacheInputs' dry.json
    

    That gives you the env vars, the lockfile hash, the root package.json hash and the global dependencies. Diff that file between two CI runs. Ninety percent of the time the culprit is one of GITHUB_RUN_ID, GITHUB_SHA, a build number, or a release identifier like SENTRY_RELEASE that you added to globalEnv months ago and forgot about.

    The fix is to move anything volatile out of globalEnv and into globalPassThroughEnv. Pass-through variables are handed to the task at runtime but are deliberately excluded from the hash, which is exactly what you want for a release tag - the build output does not meaningfully depend on it, and if it does, that is a different conversation.

    Also check whether you have "globalEnv": ["*"] anywhere, because that is a foot-gun that puts the entire CI environment in the hash and guarantees a permanent miss.

    109
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @packlist_pen · 4d ago

      It was SENTRY_RELEASE. Someone added it two years ago when we were on turbo 1.x and it contains the run number, so it is unique per build. Moved it to globalPassThroughEnv and CI is now 12/14 hits, 38 seconds. Thank you, that was going to cost me another day.

      28
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @whetstone_wu · 4d ago

      Worth adding a CI check so this cannot regress: run --dry=json twice in a row at the start of the pipeline and fail if globalHash differs. Ten lines and it catches the next person who adds a timestamp to the environment.

      24
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @whetstone_wu · 2d ago · 2 replies

    Since you already found the env var, some other things in the same family that bite people, so this thread is useful later:

    • Files present locally but not on CI. Task inputs default to everything tracked in the package directory. If you have a .env.local or a generated file that exists on your machine and not on CI, the hashes differ in the opposite direction and you get misses only locally.
    • Cache disabled silently. If TURBO_TOKEN is wrong rather than missing, some setups log a warning that scrolls past and then run uncached. Grep the log for the word cache before trusting that it is enabled.
    • Different --filter between local and CI. Filtering changes the task graph, and a task that runs with different dependencies has a different hash.
    • Lockfile churn. Any package manager that rewrites the lockfile during install invalidates the global hash on every run. Frozen lockfile in CI, always.
    72
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @kraut_corner · 4d ago

      The silent-disable one got us for a month. We had rotated the token, CI kept building fine, and nobody noticed the pipeline had gone from 45s to 5 minutes because it crept up gradually. Now we assert on the summary output - if hits are below a threshold on a no-op commit, the job fails.

      25
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @durable_ines · 4d ago

    One more that is easy to miss: package manager version drift. If your machine runs pnpm 10.4 and CI runs 10.1 and the lockfile format or resolution differs even slightly, the lockfile hash changes and everything downstream misses.

    Pin it in the root package.json with "packageManager": "[email protected]" and let corepack enforce it. Costs nothing, removes an entire category of "works on my machine" from the build cache specifically.

    46
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @petra_lindqvist · 4d ago

    For next time - turbo run build --verbosity=2 prints the hash for each task as it computes it, so you can see which task diverges first. If the global hash is stable and only one package misses, the answer is in that package's inputs rather than the environment.

    23
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @late_stage_phd · yesterday · 2 replies

    Remote cache only actually works if you are running on Vercel's own CI runners. Elsewhere it uploads but the read path is restricted, which is why you see artifacts going up and nothing coming down. It is a platform lock-in thing.

    12
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @dartfit_dana · yesterday

      This is not true. Remote cache works from any runner with TURBO_TOKEN and TURBO_TEAM set, and plenty of people run it against a self-hosted cache server with TURBO_API pointed elsewhere. The OP's own logs show uploads succeeding, which would not happen if the runner were restricted.

      10
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @probe_to_ground · 2d ago

    Every build cache problem is one of: something in the hash that should not be, or something not in the hash that should be. The second one is worse because it ships.

    6
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report