The Judge Problem: Agentic Verifier is Rotting in Production
summary
Argues verifiers silently fail: second-model judges produce no alerts while emitting optimistic scores, driven by measurable biases like position bias and verbosity bias and a strong self‑preference for text they predict. Human reviewers rubber‑stamp, and self‑verification often performs ritual checks with no causal effect. Recommends independent panels of different model families, continuous spot‑checks, and cost‑aware routing as a multi‑armed bandit (practical: Thompson sampling) because serious verification can double or triple inference cost but prevents expensive, silent failures.
AI-written summary · about
chapters (14)
- 0:00Meet Yaroslav
- 0:48Why Judges Rot
- 2:24Silent Failure Mode
- 3:54Biases In LLM Judges
- 5:49Humans Rubber Stamp
- 7:21Weak Judge Myth
- 8:42No Certificate No Asymmetry
- 9:55Limited Reasoning Bias
- 11:20Self Verification Is Fake
- 14:33Panels Not Voting
- 17:32Bandit Judge Routing
- 21:04Economics Of Verification
- 23:33Design Rules Checklist
- 25:32Final Warning And Close
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
19mFuzzing in the GenAI Era
34mScaling Trust
26mWeaver: Shrinking the Generation-Verification Gap with Weak Verifiers
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
28mTrustworthy LLMs: Vibe checks are not all you need
11mThe Unbearable Lightness of (Agentic) Evaluations