The judge model can change who gets a pass.
Published Oct 4, 2026Read first
What it saysThe authors crossed ten reasoning models with different prompts and scoring scales. Average answer-grading accuracy was 96.5%, yet switching models changed the share of errors that were overly forgiving by as much as 56.1 percentage points. Human ratings anchored the sentiment and toxicity tests.
Why it mattersTwo judges can report similar accuracy while sending very different mistakes to production. An aggregate score can hide whether a checker is too harsh or too forgiving.
Try thisRun two judge models on the same trusted examples. Compare false acceptances and false rejections, not just accuracy.
How to use it
Suggested steps
- Build a small set with independently checked good and bad outputs.
- Hold the examples fixed while changing one setting at a time.
- Record error direction, item-level verdict changes and human agreement.
| Same test set | Inspect separately |
|---|---|
| Judge A and Judge B score alike | Do they fail on the same items? |
| Both miss ten cases | Which accepts bad work? |
| Both pass calibration | Does a new task change the pattern? |
Keep in mind The study covers sentiment, toxicity and answer correctness, not tool-using agents. Its answer-correctness labels partly use consensus from the judges being tested, weakening that result.
A quick check for yourself
Can two judges with similar accuracy create different release risks?
Show answer
Yes. One may accept bad work while the other rejects good work.
Source & why it’s here
Read first: the most transferable evidence this week that judge accuracy alone can hide an important release risk.