AI-Powered Root Cause Analysis at Scale: From Theory To Production...
summary
Nubank builds a Python orchestrator that runs in Kubernetes, using ChatGPT-4 then GPT-5, Prometheus, Honeycomb, GitHub and a Blueprint API to do autonomous RCA across a 120M+ customer platform. On‑premises filtering, statistical compression and sub‑agent summarization shrink terabyte‑scale telemetry so the Prometheus MCP’s 23,000 metrics to ~14 relevant points before LLM reasoning. A pluggable discovery layer using the Model Context Protocol and Expert Guides delivers flows/runbooks; deployed in 2 months by two engineers, investigations cost < $0.20 and run in production.
AI-written summary · about
chapters (8)
- 0:00AI-powered RCA at Nubank
- 4:56Nubank’s environment challenges
- 10:08Build versus buy decision
- 13:38SRE agent architecture
- 16:05Solving the Dory problem
- 19:14Context size handling
- 20:20Six months in production
- 22:26Q&A
related talks
32mGenerative AI for Observability: Automating Root Cause Analysis in Modern IT Systems
17mSponsored keynote: Can AI handle the complexity of Kubernetes and cloud-native environments?
38mBuilding AI responsibly with a GRC strategy
41mGetting real DevOps ROI with your Generative AI
34mSolving Kubernetes Mysteries with K8sGPT: Your AI-Powered Troubleshooting Companion
14mImproving Cloud Reliability at Scale Using Gen AI
43mCONTEXT ENGINEERING FOR SELF-HEALING AI SRE
24mFrom First Experiments to AI Agents: Adoption Experience and Future Challenges