The right number can cite the wrong fact.
Published Oct 6, 2026
What it saysThe study held arithmetic and numbers fixed while moving citations between table cells with the same value. The checker sometimes accepted the wrong financial role and sometimes withheld valid alternatives. On 36 fresh source pages, 25 of 53 reviewer-supported citations were withheld at the chosen cutoff.
Why it mattersMatching a value does not prove that its source supports the claim. Period, unit and business role need their own check, while valid alternative evidence must not be rejected blindly.
Try thisTake one passing evidence-based answer. Swap its citation for a same-value cell with a different role, then test whether your checker notices.
How to use it
Suggested steps
- Check calculation correctness and evidence support as separate events.
- Add same-value wrong-role swaps plus valid alternative controls.
- Measure bad citations accepted and good citations withheld at each cutoff.
| Keep fixed | Change or compare |
|---|---|
| Answer and arithmetic | Citation points to same number, wrong role |
| Answer and arithmetic | Citation expresses the same valid fact |
| Checker decision | Bad accepted vs good withheld |
Keep in mind The fresh test used 36 selected finance pages and one reviewer, with no inter-reviewer agreement. It measures a component, not population failure rates.
A quick check for yourself
Does finding the same number prove a citation supports the claim?
Show answer
No. The period, unit and meaning of the cited cell must also match.
Source & why it’s here
Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge (opens in a new tab)
Third: a precise stress test for source-grounded agents, with a clear warning about review and sample size.