How do I A/B two coding models on the same task without fooling myself?

Hold constant: same repo commit, same starting context, same prompt file read from disk rather than retyped, same tools available, same reasoning setting. Vary exactly one thing.

Run each task twice per model. Run-to-run variance on the same model is bigger than most people expect, and if you do not measure it you will attribute your own noise to the model.

Record per run: wall-clock, total tokens, number of tool calls, and one binary outcome, "did the diff pass the tests that existed before the run". The binary is what makes the whole thing comparable later. Everything else is diagnostic colour.

Grade blind, and grade the next day. Strip model names from the transcripts and diffs, give the files neutral IDs, shuffle them, then score.

My hit rate for guessing which model produced which diff, on models I was sure I could tell apart, was barely better than a coin flip once the names were gone. That result alone changed how much I trust my own impressions from a normal working day.

Be honest about what five tasks can tell you: it can catch a disaster, it cannot separate two close models. If model A goes 5/5 and B goes 4/5, you have learned essentially nothing.

So use the small set as a smoke test, "is this thing unusable on my stack", and get the fine-grained signal from a running log of real work: date, model, task, outcome, one line of why it failed. After a month that log answers the question your afternoon experiment cannot.

Measure cost per completed task, never cost per token. A model that is three times cheaper per token and needs four attempts is the expensive one, and it also cost you the attention of four review cycles.

Two things that quietly break this calculation: fast or priority variants of the same model name carry a different rate entirely, and retries after a failed run are usually not counted by whoever is telling you their per-token figure.

Do not run the eval on your nicest repo. Run it on the one with the 900 line file, the two competing conventions and the tests that were commented out in 2023. That is where the differences between models are visible, and it is also where you actually work.