Bo

Hiding the labels costs nothing and changes results more than people believe until they try it.

One addition that costs nothing and changes the result: hide which output came from which model before you judge them.

Knowing that one came from the new release is enough to make it look better. This is not a small effect and it is exactly what has been happening to you on release day - the newer one feels better, and feels is doing all the work.

So: two columns, shuffled, labels hidden. Score them, then reveal.

A couple of things that make the scoring useful:

Score against your one-line expectation, not against each other. Better than the other one is a weaker claim than good enough for this case.

Record why, briefly, when one loses. The pattern in the failures is worth more than the total. Ten losses that are all the same kind of mistake is a prompt problem, not a model problem.

Count regressions separately from improvements. A model that is better on average and worse on your three hardest cases may be the wrong choice, and an average score hides that completely. In practice the regressions decide the switch more often than the improvements do.

And if the two are close, stay where you are. Switching has a cost, re-tuning prompts, new failure modes, unfamiliarity - and close is not a reason to pay it.