I have 12 test cases. Every prompt change I make moves the pass rate by 8 or 16 points, which even I can tell is just one or two cases flipping rather than a real signal.
So how many do people actually run with? Is 500 the target, is it 50, and does anyone have a rule for when the number starts being trustworthy? I would rather hear what teams genuinely maintain than what an ideal methodology would suggest.
The number nobody wants: you need per-category counts, not a total. 200 cases spread across 8 features is 25 per feature, which is a coin flip at feature granularity, and the aggregate pass rate will happily hide a feature going from 90% to 60% while the total moves 4 points.
Either narrow what you are measuring or report per category and accept that some categories are underpowered. One honest number per feature beats one dishonest number overall.
Start by labelling 50 real production traces before you write a single synthetic case. You get the actual distribution of what people ask, which is never what you would have invented, and you will find two or three failure modes you did not know existed. That exercise has never once failed to surprise me.
Did this over two evenings. Found that 30% of real questions are a category I had zero test cases for, and my 12 cases were all things I personally found interesting.
Make adding a case cheap or your set stays at 12 forever. Ours is a folder of JSON files, one per case, and adding one is copy, paste, edit two fields. If adding a case involves a UI, a review and a naming convention, nobody will do it on the Friday afternoon when they actually have the failing example in front of them.
The rule that grew ours without anyone ever scheduling time for it: nothing gets marked fixed until there is a case in the set that fails before the fix and passes after. No exceptions, including for the founder.
We went from 14 cases to 140 in about eight months and every one of them corresponds to something a real person complained about. That also means the set is automatically weighted towards the things that break, which is the only weighting worth having.
12 that you have read carefully beats 500 generated by a model that you have never read. I have audited synthetic eval sets where roughly a third of the reference answers were wrong, which means the harness was actively punishing correct behaviour.
Generating candidate cases and then having a human accept or reject each one is the version that works. Generating the answers too is where it goes wrong.
However many it takes before you stop shipping prompt changes on vibes. For most teams that number is smaller than they fear and larger than 12.
@vesting_cliff · 9mo ago · 3 replies
12 is not nothing. 12 is a smoke test and you should keep it as one, because it runs in forty seconds and it catches the catastrophic changes. What it cannot do is detect a small regression — one flipped case is eight points, exactly as you noticed.
The shape teams actually maintain, roughly:
But the count matters much less than the composition. 200 cases that are all happy-path questions tell you almost nothing. 60 cases where 25 of them are things that actually broke in production is a real instrument. Build the set out of your bug reports and support messages, not out of imagination.
And be honest about what you can detect. At an 80% pass rate with 100 cases you can see a 10 point regression clearly and a 3 point one not at all. Decide which one you care about, then size for that, and stop treating the pass rate as a number with three significant figures.
Reply
Report
@seam_ripper_joy · 9mo ago
Three tiers rather than one set is the reframe I needed. I have been trying to make one collection serve as both the fast check and the release gate and it was bad at both.
Reply
Report
@pomodoro_pat · 9mo ago
"Build it out of your bug reports" is the whole answer. Every eval set I have seen built in a vacuum was abandoned within two months, because nobody believed the cases represented anything.
Reply
Report