Taming Rogue AI Agents with Observability-Driven Evaluation
summary
Claims a repeatable observability pipeline that logs per-step traces, scores them with a stronger LLM, and surfaces actionable fixes; demo chatbot required three steps to return a balance so evaluations flag the problem with action completion and action advancement. Framework runs on production traces (millions/day) and recommends testing ~10,000 samples with an expensive scorer LLM or a custom-trained model, produces human-reviewable suggestions (add explicit system instructions to fetch all accounts), and plugs into CI/CD with CLHF for continuous tuning.
AI-written summary · about
chapters (7)
related talks
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
41mFrom Lab To Life: Practical AI System Evaluation
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
25mFrom Chaos to Control: Building Enterprise-Grade LLM Agents That Actually Work
23mContinual Learning for AI Agents: From Failures to Durable Improvements
30mTutorial: Building an Observable AI Agent With an Open Source Stack
36mAdvanced Evaluation Techniques for AI Agents