A correct verdict can rest on invented reasoning.
Published Oct 5, 2026
What it saysTwo security experts found fabricated or unverifiable claims in about 60% of correct vulnerability verdicts. The proposed checker turns free text into a structured record, then applies deterministic and semantic checks across eight failure types. It exposed 87% of errors missed by a free-text AI judge.
Why it mattersA final pass can be lucky. A compact record of evidence, state changes and assumptions makes hidden gaps testable without trusting persuasive prose.
Try thisFor one critical workflow, require a short structured record of evidence, actions and state changes. Check the fields separately from the final answer.
How to use it
Suggested steps
- Define the few facts and transitions a valid solution must expose.
- Use code checks for exact fields; reserve AI judgment for meaning.
- Inject known record errors and measure which checks catch them.
| Layer | Check |
|---|---|
| Final verdict | Is the answer right? |
| Reasoning record | Are evidence and state changes explicit? |
| Targeted checks | Which exact claim or transition failed? |
Keep in mind This is a security-specific study over code slices. Some checks miss implicit operations and aliases, and the mutation tests may not capture every real reasoning failure.
A quick check for yourself
Why is a correct final verdict not enough?
Show answer
It may rest on invented evidence or an impossible path that fails on the next case.
Source & why it’s here
Second: a strong method for separating a correct outcome from the quality of the path used to reach it.