turborepo 2.x remote cache misses on every ci run but hits locally Turborepo
14 packages, pnpm workspaces, turbo run build with remote caching enabled. On my machine the second run is 100% cache hit in about 400ms. On CI it is 0/14 every single time, 6m41s, on a commit that changed one README.
Things I have verified:
TURBO_TOKENandTURBO_TEAMare set on CI, and the run summary does say it is uploading artifacts afterwards. So it can talk to the cache, it just never finds anything.- same turbo version locally and on CI (checked with
turbo --version). - same Node major.
--dry=jsonon CI shows a differentglobalHashon two consecutive runs of the same commit.
That last one seems like the actual clue - if the global hash changes between two identical runs then obviously nothing can ever hit. I just cannot work out what is feeding into it that is different each time.
What is in the global hash that I am not thinking about?
@dartfit_dana · 3d ago · 3 replies
Two identical runs producing different global hashes means an environment variable is in the hash and its value changes per run. That is basically the only thing it can be.
Dump the inputs and read them:
That gives you the env vars, the lockfile hash, the root package.json hash and the global dependencies. Diff that file between two CI runs. Ninety percent of the time the culprit is one of
GITHUB_RUN_ID,GITHUB_SHA, a build number, or a release identifier likeSENTRY_RELEASEthat you added toglobalEnvmonths ago and forgot about.The fix is to move anything volatile out of
globalEnvand intoglobalPassThroughEnv. Pass-through variables are handed to the task at runtime but are deliberately excluded from the hash, which is exactly what you want for a release tag - the build output does not meaningfully depend on it, and if it does, that is a different conversation.Also check whether you have
"globalEnv": ["*"]anywhere, because that is a foot-gun that puts the entire CI environment in the hash and guarantees a permanent miss.Reply
Report
@packlist_pen · 4d ago
It was
SENTRY_RELEASE. Someone added it two years ago when we were on turbo 1.x and it contains the run number, so it is unique per build. Moved it toglobalPassThroughEnvand CI is now 12/14 hits, 38 seconds. Thank you, that was going to cost me another day.Reply
Report
@whetstone_wu · 4d ago
Worth adding a CI check so this cannot regress: run
--dry=jsontwice in a row at the start of the pipeline and fail ifglobalHashdiffers. Ten lines and it catches the next person who adds a timestamp to the environment.Reply
Report