Build Evals That Actually Matter
summary
Claims a simple cause and shows concrete fixes: offline evals hit 90% offline pass yet production fails because the synthetic user is too polite, so Lyft fine-tunes a fine-tuned LLM user simulator on millions of real transcripts that catches regressions the old eval missed. The deployed system handles agent resolves ~1/3 of issues at scale, uses an calibrated LLM-judge to human agreement, YAML-driven harness, trace routing, and a continual-learning loop that feeds real failures back into tests.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:31End-to-end eval pipeline
- 2:14Offline eval for AI agents
- 7:21Why eval systems fail
- 7:43Designing robust offline eval
- 11:21Synthetic data challenges
- 13:24User simulation for evals
- 17:41Limitations of LLM judges
- 19:47Actionable eval metrics
- 21:34Validating LLM judges
- 23:43Criteria drift in evals
- 25:08Statistical rigor in scores
- 26:55Eval anti-patterns
- 28:19Error analysis loop
- 29:21Tracing in evaluation
- 31:07Closing eval loop
- 32:56Future eval harness plans
- 34:24Agent-facing eval framework
- 36:55Summary & Q&A
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
47mBuild Trustworthy AI Agents powered by Evals
49mEvals 101
06mWhy should anyone care about Evals?
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
19mThe maturity phases of running evals
36mAdvanced Evaluation Techniques for AI Agents