How do you get the list of files changed in a pipeline run, reliably?

The reason your script fails intermittently is that "what changed" has no single answer - it depends on what the run is for, and the runs are not all the same kind.

The two questions people conflate:

• What did this push add? Compare the previous head to the new head. Correct for a push to a long-lived branch, meaningless for a new branch, since there is no previous head.
• What does this branch change relative to where it will land? Compare against the merge base with the target branch. This is the right question for anything reviewing a proposed change, and it is stable across re-runs and force pushes, which the first one is not.

Almost everybody wants the second and writes the first. That is the bug.

Get the merge base explicitly rather than assuming a parent:

git fetch origin main
base=$(git merge-base HEAD origin/main)
git diff --name-only "$base" HEAD

Worth knowing that most CI systems have built-in path filters, and for the common case they are better than any script.

A rule saying this job only runs when files under a given path changed is evaluated by the platform, handles the branch and merge cases itself, and cannot drift out of sync with your diffing logic.

Use the script only for what the filters cannot express, passing the actual file list to a tool, or fanning out over changed directories.

For a repository with several packages, the further step is a build tool that understands the dependency graph. Path filters know that a file changed; a build graph knows that changing it means three packages need rebuilding and the other twelve do not. That distinction is where the real time is saved, and no amount of diffing gets you there.

The second cause of intermittent failure is the one nobody sees coming: CI clones are shallow by default.

A shallow clone has no history, so merge-base cannot find one and the diff either fails or returns everything. It works on some runs and not others depending on how much history happened to be fetched, which produces exactly the "fails confusingly a few times a month" you describe.

Either fetch enough depth, or unshallow before diffing:

git fetch --unshallow 2>/dev/null || true

The cost is a slower checkout on big repositories, which is why the default is shallow. Fetching just the target branch with enough depth is the middle ground.

Also handle the first-run case explicitly. When there is genuinely nothing to compare against, the safe default is run everything, not run nothing. A pipeline that silently skips its tests because a diff came back empty is worse than a slow one.

One caution about skipping on unchanged paths that catches teams eventually: a required check that is skipped may never report.

If a merge is blocked until a job passes, and the job is skipped because nothing relevant changed, the merge can hang forever waiting for a status that will never arrive. Most platforms have a way to report a skipped job as successful, and it is worth configuring deliberately rather than discovering it on a Friday.