The pipeline started failing with no changes to the repository — where do I even begin looking?
Our build has been green for weeks. Nobody merged anything. This morning it fails, and it fails identically on re-run and on branches that have not been touched in a month.
Re-running the last successful commit — the exact same commit that was green — also fails now. So it is definitively not our code.
The error itself is not obviously informative: a dependency install step fails partway through.
I know "something external changed" is the answer, but that is a big space. Is there an order to search it in?
@reproducible_rui · 4h ago
There is an order, and the fact that re-running a previously green commit now fails is the most valuable piece of information you have. It proves the input that changed is not in your repository, which eliminates most of the search space immediately. Keep that test — it is how you will verify the fix too.
Work through these, roughly in order of how often they turn out to be the cause:
1. An unpinned dependency published a new version. By far the most common. Any dependency specified as a range rather than an exact version, in any lockfile-free layer, can change under you overnight. That includes the ones people forget are dependencies: build plugins, linters, type definitions, and the transitive dependencies of your dependencies.
The tell is that the failure is in an install or build step, which matches yours exactly. Compare the resolved versions in the failing run against the last green run — if your CI logs the resolution, this is a two-minute diff and it usually names the culprit outright.
2. The base image moved. If your Dockerfile or job uses a floating tag, that tag points somewhere else today. A minor version bump of the base image can change the system library versions, the default user, the package manager's behaviour or the installed toolchain.
3. The runner image was updated. Hosted CI providers update their runner images continuously, and those images carry preinstalled toolchains. A new default language version, a removed system package or a changed default path breaks builds that were relying on the old one without knowing it. Providers publish changelogs for these; check whether one landed in the window.
4. A credential or certificate expired. Registry tokens, deploy keys, signing certificates, cloud credentials. This produces a very confident-looking failure that has nothing to do with what it says, particularly when a private registry returns a 401 and the package manager reports it as "package not found".
5. Rate limiting or a quota. Anonymous pulls from public registries are rate limited, and you cross the threshold as the team grows or as retries multiply. Intermittent at first, then constant. Frequently misdiagnosed as flakiness for weeks.
6. An upstream package or repository was removed, renamed or deprecated. Rarer, but it happens, and the error is usually clear once you look at the right line rather than the last line.
How to narrow it fast. Get the full log of the last green run and the full log of the first red run and diff them. Not the errors — the whole thing. Tool versions, resolved package versions, image digests, base image ids. Something in there is different, and diffing the logs finds it faster than reasoning about it.
If you do not retain old logs, start. This exact situation is what they are for.
Reply
Report