From Chaos to Confidence: How SREs Can Leverage 50 (and Counting) Failure...
summary
Built a sandboxed Kubernetes evaluation called IT Bench that replays 50+ production-inspired failure scenarios using Helm, Ansible, an open observability stack and OTel. Benchmarks run LLM agents (Gemini, Claude, others) for diagnosis and mitigation and report a sobering 13.8% success rate for current models; naive agents hit ~14% diagnosis and ~11% mitigation, while a topology-guided approach boosts pass@3 toward 95% on a snapshot.
AI-written summary · about
chapters (11)
- 0:00AI in SRE and incident response
- 1:43AI claims in incident analysis
- 4:54Lessons from AI experimentation
- 6:58Introducing IT Bench framework
- 9:47Evaluating AI agents with IT Bench
- 13:35LLM-assisted incident reproduction demo
- 17:04AI agent performance metrics
- 20:17Benchmark results and improvements
- 23:06Community collaboration on IT Bench
- 24:28Future of AI in SRE
- 25:11Q&A
related talks
40mBeyond Static Benchmarks: Chaos Based Agent...
17mSWE-rebench: Lessons from Evaluating Coding Agents
31mObservability in the Matrix: Choosing the Right AI Agent To Escape the Chaos
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
20mPremiere
33mStrategic AI Agents for K8s SRE: Beyond Ad-hoc Prompting
36mSRE'ing the SRE AI Agents
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents