From Self-driving to Autonomous Voice Agents
summary
Claim: apply Waymo-style evaluation to voice agents by using large-scale simulation and probabilistic, reference-free metrics. Built a CI/CD loop with presubmit/postsubmit checks, live monitoring, and a tool called metric studio that calibrates LLM judges with human labels. Shows scaling pain points — systems work for 10 conversations but break at 10,000, 80% of postdeploy effort becomes professional services — so repeat failures are re-simulated 100 times to measure if a bug is a coin flip or a 99/100 deterministic break.
AI-written summary · about
chapters (9)
- 0:00Introduction to voice agents
- 1:49Trust and scaling challenges
- 3:55Lessons from self-driving
- 6:41Constant evaluation loops
- 9:28Realism in simulations
- 12:18Building an eval strategy
- 15:26Benchmarking and testing
- 17:33Continuous monitoring
- 18:59Future of voice AI
related talks
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
33mEvaluating Your AI Agent
43mTesting Agentic AI Applications: Beyond Traditional QA
46mEnhancing Agent Reliability Through Evaluation-Driven Development
56mAI Agents That Don’t Suck: How to Build Ones That Actually Drive Business Value
42mRobust Testing Strategies for AI-Driven Applications
37mAI agents testing How to evaluate the unpredictable
14m15 Months of building AI agents in 15 minutes