The Unbearable Lightness of (Agentic) Evaluations
summary
Claims that agentic evaluation is fragile and nonreproducible, and that open tooling is required to own scores. A simple proxy called Kodak contamination proxy measures log-probability shifts when an in‑context example is added and discriminates seen vs unseen data; six open‑source models (three 2025, three 2026) show math improves while agentic and coding robustness does not. A common failure is action‑thought disconnect (model refuses in text but executes a dangerous chmod), motivating Nemo Evaluator SDK as an open evaluator stack.
AI-written summary · about
chapters (11)
- 0:00Agentic evaluations intro
- 1:03AI evaluation challenges past
- 2:06AI evaluation progress since
- 2:50Agentic model challenges
- 4:17Kodak data contamination detection
- 5:43Math and agentic task benchmarks
- 6:24Action thought disconnect
- 7:50Beyond leaderboard evaluation
- 9:18Agentic evaluation practicals
- 10:21Nemo Evaluator SDK
- 11:04Conclusion on evaluation
related talks
20mAgentic Evaluations at Scale, For Everybody
46mEnhancing Agent Reliability Through Evaluation-Driven Development
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications- 02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
20mFive hard earned lessons about Evals
22mCracking the Code: Deciphering Evaluation Essentials for RAG
27mThe Judge Problem: Agentic Verifier is Rotting in Production
26mAI Observability and Evaluation: Ensuring Trustworthy Intelligent Systems