Grace

Add it to your tests. One case: run a batch containing prompts of clearly different lengths, and assert the output for each row matches the output you get running that same prompt alone.

It is a cheap test and it catches this whole family — padding side, position ids, per-row slicing — the moment someone changes the batching code. Without it the failure is invisible in aggregate metrics and only shows up as users saying the thing feels worse.

Once you have found it, the fix is not just "pin that one thing". The lesson is that your build was not reproducible, and it will happen again with a different dependency.

What actually stops it recurring:

• Commit a lockfile and install from it in a mode that fails rather than updating. Every ecosystem has a distinct command for this: install exactly what the lockfile says, error if it cannot. Using the ordinary install command in CI is how a lockfile silently stops protecting you.
• Pin base images by digest, not by tag. A tag is a moving pointer; a digest is immutable. Slightly annoying to update, completely deterministic.
• Pin the runner image version if your provider allows it, rather than the rolling label.
• Cache or mirror your dependencies so you are not exposed to upstream availability on every build.

All of that costs an afternoon, and you get it back the first time this would have happened again.

Budget for it, though: pinning means you now own the updates. A renovation bot that opens pull requests for pinned versions is what makes it sustainable, because the alternative — pinning and never updating — is a different problem in twelve months.