Are We All on the Same Page? Let's Fix That
summary
Claim: a webhook-style handler called Adaptive Paging ingests alerts, queries recent OpenTelemetry traces, traverses causal child spans looking for status.code=error, and pages the owning service instead of the static alert owner. Built as trace-driven heuristics plus simple scoring for parallel failures, it requires full instrumentation and a service registry (Backstage-style) and reduces “Christmas tree” multi-team alerts by >99% in practice at Delivery Hero’s multi-platform environment. It works, with caveats around missing instrumentation and simultaneous independent failures.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 2:28Monoliths to microservices
- 5:57Microservice monitoring challenges
- 8:46Christmas tree alerting
- 10:12Symptom-based alerting
- 13:24Adaptive paging introduction
- 14:51Distributed tracing primer
- 19:23Symptom-based alerting rules
- 20:47Adaptive paging in practice
- 23:15Adaptive paging examples
- 24:18Adaptive paging challenges
- 25:42Incentivizing tracing adoption
- 26:45Service ownership mapping
- 27:49Why the industry hasn't improved yet
- 30:21Industry progress and vendors
- 31:24Q&A
related talks
39mAre We All on the Same Page? Let's Fix That
50mApplied Alerting Philosophy
36m"Alert overload: How to adopt a microservices architecture..."
41mA Theory and Practice of Alerting with Service Level Objectives
35mDon't page me! How we limit pager noise at New Relic
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
27mBuilding a Scalable Monitoring System
16mHuman Centered On Call Practices