my llm judge gives 4.6 out of 5 to answers i deliberately broke llm-as-judge

Your judge prompt is bad in a very standard way, and absolute 1-5 scales are the cause. With no anchor the model settles into "this looks like an answer, 4.5" and stays there forever.

Three changes, in order of impact:

• Pairwise instead of absolute. Show it A and B, ask which is better and why. Models are far more reliable at comparison than at scoring. Randomise which one is presented first, because position bias is real and large enough to reverse conclusions.
• Binary criteria instead of a scale. "Does every factual claim appear in the provided context — yes or no." "Is the answer complete — yes or no." Five binaries carry more information than one 1-5 and they are individually debuggable.
• Reasoning before verdict, and make the verdict a single parseable token at the end.

Then the step nearly everyone skips, which is the one that makes the rest mean anything: hand-label 50 examples yourself and measure how often your judge agrees with you. Below about 80% agreement your eval numbers are decoration. You already have the beginnings of this — your deliberately broken examples are labels.

Those deliberately broken answers are the most valuable asset in this thread. Freeze them as a labelled regression set — known bad, with the reason. Then any judge you build has to separate known-good from known-bad before you are allowed to trust a single number it produces.

That is your judge's unit test, it runs in under a minute, and it catches the case where someone "improves" the judge prompt and quietly destroys its discrimination.

For fabricated citations specifically, do not use a model at all. That is a string containment problem: for each cited span, check it actually appears in the retrieved context. Deterministic, free, instant, correct every time.

Save the judge for things that genuinely need judgement, like whether an answer addressed the question that was asked. Every check you can turn into an assertion is a check you never have to calibrate again.

The other half of why everything scores 4.6: your judge has never seen a bad answer, so it has no idea where the bottom of the scale is. Put two examples in the prompt, one clearly poor with the reasoning spelled out and one clearly good, and the distribution opens up immediately. Anchoring beats instruction here.

4.6 out of 5 for everything including the broken ones is not an eval, it is a customer satisfaction survey.

Related question since this thread is well informed: is anyone using a cheaper model as the judge, and does it hold up? The judge calls are becoming a real line on our bill.