Building an Observability Agent for Rapid Root Cause Analysis using Prometheus metrics
summary
A Prometheus-first observability agent is built that queries Prometheus metrics, pulls logs and traces from OpenSearch via Prometheus MCP server and OpenSearch MCP server, and uses a plan‑execute‑reflect workflow with claude-3.7 for analysis. A demo on a hotel telemetry shopping cart finds a persistent cart_failure feature flag breaking empty‑cart operations and recommends disabling it, the long‑running agent takes ~5 minutes to complete, and reported reliability is roughly 60–70% with visible tool traces for validation.
AI-written summary · about
chapters (15)
- 0:00Observability agents for RCA
- 1:25RCA workflow challenges
- 3:53Agents for automated RCA
- 5:58Agent development workflow
- 7:44Agent architecture and setup
- 8:48Live RCA agent demo
- 15:16Plan-execute-reflect agent
- 17:45Agent accuracy challenges
- 20:16Key takeaways and future apps
- 22:45Q&A: Memory usage
- 24:29Q&A: Production success
- 25:52Q&A: Data filtering
- 27:42Q&A: MCP security
- 29:05Q&A: Noisy systems
- 31:53Q&A: Cost efficiency
related talks
56mFull-stack observability and application monitoring with
37mBuilding Modern Observability Architecture With OpenSearch, OS Ingest...
27mAlert Fatigue to Action: Observability as the Backbone of Practical AIOps with OpenSearch
24mFrom Monitoring to Observability: 10 Things to Do with Prometheus in 2020
18mBuilding an APM with OpenTelemetry and OpenSource
55mBeginner's guide to observability
28mScaling Observability for Real-Time Personalization Engines
50mApplication metrics with Prometheus