A new model ships and I switch on release day, how do you actually test whether it is better for your task in an afternoon?

One addition that costs nothing and changes the result: hide which output came from which model before you judge them.

Knowing that one came from the new release is enough to make it look better. This is not a small effect and it is exactly what has been happening to you on release day - the newer one feels better, and feels is doing all the work.

So: two columns, shuffled, labels hidden. Score them, then reveal.

A couple of things that make the scoring useful:

Score against your one-line expectation, not against each other. Better than the other one is a weaker claim than good enough for this case.

Record why, briefly, when one loses. The pattern in the failures is worth more than the total. Ten losses that are all the same kind of mistake is a prompt problem, not a model problem.

Count regressions separately from improvements. A model that is better on average and worse on your three hardest cases may be the wrong choice, and an average score hides that completely. In practice the regressions decide the switch more often than the improvements do.

And if the two are close, stay where you are. Switching has a cost, re-tuning prompts, new failure modes, unfamiliarity - and close is not a reason to pay it.

The afternoon version is a fixed set of real inputs and a spreadsheet. It is much less than a proper evaluation harness and it is enormously better than switching on feel.

Collect twenty to thirty real cases from your actual traffic. Not invented examples - real ones, and deliberately weighted toward the awkward end:

• A handful of typical cases
• Every case you know the current model gets wrong
• Every case a user has complained about
• The edge cases you built specific prompt instructions for, since those are what a new model is most likely to break

That last group is the one that saves you. Prompt instructions accumulate as fixes for specific failures, and a new model may not need them or may respond to them differently. Those are your regressions.

Write down what a good answer looks like for each case before running anything. Not a full rubric, one line. Doing this first is what stops you from grading generously afterwards.

Run all cases through both models, save the outputs side by side.

Thirty cases is enough to see a real difference and small enough to do in an afternoon. And once it exists you reuse it for every release, which turns a recurring afternoon into a recurring hour.

Keep the twenty cases in the repository. The value compounds, because next release you already have the harness.

It feels better is not nothing, it is just not evidence, and it is indistinguishable from novelty.

The two numbers people forget until they matter: cost and latency on your actual inputs.

Both are easy to measure while you are running the comparison and both are frequently the deciding factor:

Cost per real request, not per token in the abstract. A model that is cheaper per token and more verbose can cost more. Measure with your prompts and your typical inputs.

Latency at the percentile your users feel, not the average. If a request sits in front of somebody waiting, the slow tail is what they experience.

Write both into the same sheet as the quality scores. Then the decision is three numbers rather than a feeling, and it is often clear immediately - a small quality gain that doubles your latency is usually not worth it for an interactive product, and is fine for a background job.

Two operational notes since you run this in production:

Pin the version you tested. Testing one thing and running another because an alias moved underneath you is the most common way this whole exercise gets wasted.

Keep the old one available for a while after switching. The regressions you did not catch surface within days from real traffic, and being able to switch back in one config change is the difference between an inconvenience and a bad week.