I wrote 30 questions with expected answers, I score them myself, and every change I make lands somewhere between 20 and 24 correct. Chunk size, reranker on or off, a different prompt — all within that range. Either everything I try is useless or my eval cannot see the difference. New to this and I do not know which.
Grow the set from real traffic instead of your own head. Every question a user asked that got a bad answer becomes a case, with the correct chunk attached. Thirty questions you wrote yourself are all in your own vocabulary and phrasing, which is precisely the distribution you do not need help with.
My questions are all grammatical sentences ending in question marks. The logs are three words, a typo and an internal acronym. Point taken.
Split the eval in two. Retrieval metrics — recall@k, MRR on a set of question-to-known-chunk pairs — are deterministic, run in seconds, cost nothing and catch the large majority of regressions. Judging the final answer is noisy and slow, so do it last and on fewer cases. Most people build the expensive half first and then cannot afford to run it.
Also normal to not need more than this. For a small internal tool, "does it still handle these 12 things people ask constantly" is a legitimate bar. Do not build an evaluation platform for a bot with 12k documents and 40 users.
If you move to a model judging the answers, do not use the same model family that generated them. It rates its own phrasing generously and you will get a lovely upward line that means nothing.
@gradschool_gary · 2d ago · 2 replies
Both, sort of, and the second one is fixable today.
With 30 items, 21 versus 23 correct is noise. You will never resolve a real 5% improvement at that sample size by staring at aggregate scores.
The fix is not necessarily more questions, it is a paired comparison. Run the old and new versions over the same 30 questions, then only look at the ones where the two answers differ. Usually that is four or five cases and you can judge them properly in ten minutes. "3 got better, 1 got worse, 26 unchanged" is actionable in a way that "22 vs 23" never is.
Reply
Report
@frosting_fixer · 4d ago
Log the retrieved chunk ids alongside every answer, not just the final text. Most "the answer got worse" turns out to be "retrieval changed", and you can see that from a diff of two id lists without reading a single word of generated prose.
Reply
Report