The Future of Evals: From LLM as a Judge to Agent as a Judge
summary
Arize reports running 100 million evals per month and sees teams run about 12 eval jobs on average with top teams at 3,800 evaluators, and argues evals must move from deterministic checks to LLM-as-judge to agent-as-judge. The claim: an agent that reads traces can discover failure modes classical rubrics miss. Arize builds Alex as a traced agent and deploys Signal that finds loops, repeated tool calls, forgotten context and inefficient trajectories and opens pull requests to fix them.
AI-written summary · about
chapters (4)
- 0:00Opening: the future of the Evals track
- 2:06Why evals got harder as agents evolved
- 3:45From deterministic checks to LLM as a judge
- 4:36Agent as a judge, and where evals go next
related talks
36mAdvanced Evaluation Techniques for AI Agents
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
46mEnhancing Agent Reliability Through Evaluation-Driven Development
35mThe Agentic AI Engineer
47mBuild Trustworthy AI Agents powered by Evals
43mEval-Driven Development: Mastering Agentic Tracing and Expert-Aligned Judges With MLf...
49mEvals 101
17mLLM Observability, Evaluation, Experimentation Platform