Ask
29
@pipeline_on_call ·

The pipeline started failing with no changes to the repository — where do I even begin looking?

Our build has been green for weeks. Nobody merged anything. This morning it fails, and it fails identically on re-run and on branches that have not been touched in a month.

Re-running the last successful commit — the exact same commit that was green — also fails now. So it is definitively not our code.

The error itself is not obviously informative: a dependency install step fails partway through.

I know "something external changed" is the answer, but that is a big space. Is there an order to search it in?

4 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @reproducible_rui · 7h ago

    There is an order, and the fact that re-running a previously green commit now fails is the most valuable piece of information you have. It proves the input that changed is not in your repository, which eliminates most of the search space immediately. Keep that test — it is how you will verify the fix too.

    Work through these, roughly in order of how often they turn out to be the cause:

    1. An unpinned dependency published a new version. By far the most common. Any dependency specified as a range rather than an exact version, in any lockfile-free layer, can change under you overnight. That includes the ones people forget are dependencies: build plugins, linters, type definitions, and the transitive dependencies of your dependencies.

    The tell is that the failure is in an install or build step, which matches yours exactly. Compare the resolved versions in the failing run against the last green run — if your CI logs the resolution, this is a two-minute diff and it usually names the culprit outright.

    2. The base image moved. If your Dockerfile or job uses a floating tag, that tag points somewhere else today. A minor version bump of the base image can change the system library versions, the default user, the package manager's behaviour or the installed toolchain.

    3. The runner image was updated. Hosted CI providers update their runner images continuously, and those images carry preinstalled toolchains. A new default language version, a removed system package or a changed default path breaks builds that were relying on the old one without knowing it. Providers publish changelogs for these; check whether one landed in the window.

    4. A credential or certificate expired. Registry tokens, deploy keys, signing certificates, cloud credentials. This produces a very confident-looking failure that has nothing to do with what it says, particularly when a private registry returns a 401 and the package manager reports it as "package not found".

    5. Rate limiting or a quota. Anonymous pulls from public registries are rate limited, and you cross the threshold as the team grows or as retries multiply. Intermittent at first, then constant. Frequently misdiagnosed as flakiness for weeks.

    6. An upstream package or repository was removed, renamed or deprecated. Rarer, but it happens, and the error is usually clear once you look at the right line rather than the last line.

    How to narrow it fast. Get the full log of the last green run and the full log of the first red run and diff them. Not the errors — the whole thing. Tool versions, resolved package versions, image digests, base image ids. Something in there is different, and diffing the logs finds it faster than reasoning about it.

    If you do not retain old logs, start. This exact situation is what they are for.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @pipeline_on_call · 7h ago

    Once you have found it, the fix is not just "pin that one thing". The lesson is that your build was not reproducible, and it will happen again with a different dependency.

    What actually stops it recurring:

    • Commit a lockfile and install from it in a mode that fails rather than updating. Every ecosystem has a distinct command for this: install exactly what the lockfile says, error if it cannot. Using the ordinary install command in CI is how a lockfile silently stops protecting you.
    • Pin base images by digest, not by tag. A tag is a moving pointer; a digest is immutable. Slightly annoying to update, completely deterministic.
    • Pin the runner image version if your provider allows it, rather than the rolling label.
    • Cache or mirror your dependencies so you are not exposed to upstream availability on every build.

    All of that costs an afternoon, and you get it back the first time this would have happened again.

    Budget for it, though: pinning means you now own the updates. A renovation bot that opens pull requests for pinned versions is what makes it sustainable, because the alternative — pinning and never updating — is a different problem in twelve months.

    27
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @vram_accountant · 7h ago

    For the immediate unblocking, before the proper fix: check whether your CI provider has a status page and whether anyone else is reporting the same thing.

    A meaningful share of "broke with no changes" incidents are the provider's, or a major registry's, and they are resolved within hours by someone else. Ten seconds of looking saves you from a deep investigation into a problem that is not yours.

    Same instinct for the ecosystem: a broken release of a very widely used package produces a lot of noise very quickly, and there is usually already a thread about it.

    20
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @adblock_realist · 2h ago

    Worth adding a scheduled build on your default branch, if you do not have one.

    It runs the same commit on a timer, so external breakage is detected by the schedule rather than by whoever happens to push next. That turns this from "my change broke the build" confusion into a clean signal that something outside changed, with a timestamp.

    It also catches the slow version: something that will break your next release but not your current one, discovered a week early rather than during a release.

    1
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report