The Failures That Don't Crash: MLOps for AI Agents
summary
Four concrete reliability patterns for production agents: shadow test against a baseline by comparing distributions (latency, first-token time, token cost, refusal rate, per-category evals), circuit breakers that use confidence thresholds and loop detection rather than error codes (max steps, schema checks, tool-call repeats), an eval harness that scores non-deterministic outputs with an LLM judge plus human-labelled checklists, and structured human oversight with reviewer chaos drills to catch automation-bias decay after ~20 minutes. Implementation notes include where to hook metrics, what failure modes to watch, and versioned test sets.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 1:50AI coding risks and eval gaps
- 3:17Shadow testing for AI agents
- 5:26Circuit breakers for AI failures
- 8:14Data assumptions and pitfalls
- 10:02Categorizing failures and eval layers
- 12:31Human monitoring limitations
- 13:56Failure drills for reviewers
- 15:21Actionable steps for tomorrow
- 16:04Conclusion and Q&A
related talks
13mMLOps: From AI Research to Production-Ready Systems
41mFrom Lab To Life: Practical AI System Evaluation
26mAre you ready for MLOps? 🫵
10mAvoiding the Oops with MLOps
43mTaming the Unpredictable: Reliability in Chaos
32mReliable Machine Learning
13mHuman-in-the-Loop MLOps: Boost Model Performance by 40%
23mContinual Learning for AI Agents: From Failures to Durable Improvements