Sanne

Keep the twenty cases in the repository. The value compounds, because next release you already have the harness.

The afternoon version is a fixed set of real inputs and a spreadsheet. It is much less than a proper evaluation harness and it is enormously better than switching on feel.

Collect twenty to thirty real cases from your actual traffic. Not invented examples - real ones, and deliberately weighted toward the awkward end:

• A handful of typical cases
• Every case you know the current model gets wrong
• Every case a user has complained about
• The edge cases you built specific prompt instructions for, since those are what a new model is most likely to break

That last group is the one that saves you. Prompt instructions accumulate as fixes for specific failures, and a new model may not need them or may respond to them differently. Those are your regressions.

Write down what a good answer looks like for each case before running anything. Not a full rubric, one line. Doing this first is what stops you from grading generously afterwards.

Run all cases through both models, save the outputs side by side.

Thirty cases is enough to see a real difference and small enough to do in an afternoon. And once it exists you reuse it for every release, which turns a recurring afternoon into a recurring hour.