What changed in
AI evaluation?

Five judges differed by only 1.6–4.4 points on task-success accuracy, but by 45–50 points on detecting undesirable side effects. High-recall screening caught over 90% of problems while routing more than 70% of traces to review.

One alignment number can conceal dramatically different failure profiles and human-review costs.

A more capable configuration increased real-data answers from 21% to 73%, yet exceeded its schema-exploration budget on 77% of traces and changed table interpretation on 41 of 50 questions.

Better-looking answers can coexist with worse process reliability—the same risk as a travel answer built from the wrong source, date or tool path.

Holding the model constant while changing its harness produced up to a 40× difference in tokens per successful task and distinct failure patterns, while pass-rate differences stayed within eight points.

Model quality is often model-plus-agent-architecture quality. Tool interfaces, retry logic and stopping behavior can dominate cost and failure type.

Across five benchmarks and 15 models, 22% of model pairs reversed ranking between benchmarks. Correlations ranged from 0.10 to 0.92.

A composite agent score can hide whether progress came from reasoning, tool use, factuality or task completion.

With identical content presented in different styles, direct judges remained style-sensitive. A style-aware step raised substance recognition from 0.504 to 0.759 and adversarial robustness from 0.554 to 0.899.

Helpfulness and completeness scores may reward polished phrasing rather than better traveler outcomes.

Rubrics derived from authentic professional deliverables substantially outperformed prompt-derived rubrics for specialized roles: 99.1% versus 78.0%. Reusing role-level rubrics reduced estimated construction work 6.7×.

Quality standards should come from real successful work, not an LLM inventing what excellent looks like.

The paper formalizes promote, reject, continue and abstain decisions for partial evaluations. Some benchmarks reached their full-run conclusion after 15–25% of outcomes; others needed 90% or more.

A small weekly sample can be decisive only when important intent slices are represented and uncertainty supports the decision.

Microsoft describes a living regression-set pipeline that maps incoming cases to roughly 30 capability dimensions, then admits, replaces or routes them for human review based on coverage and quality.

It avoids continually adding production failures until the eval set becomes slow, redundant and expensive to triage.