Building a Scalable Monitoring System
Talkat SREcon19 Europe/Middle East/Africa · on Wednesday, 2019-10-02· at 12:00· Track 1: The Liffey B·
summary
Company consolidates six disparate monitors (New Relic, Honeybadger, PagerDuty, LastAlert, StatusCake plus cron/dashboard checks) into Datadog as a single aggregator, keeps existing integrations, and gains timeboxed mute controls and alert history. All alerts must require action so false positives go away; mutability prevents noisy repeats during incidents. Onboarding shrinks to a single page and three steps, on-call burnout drops, and developer buy-in grows — alerts rise from ~30–40 to over 90 contributed monitors as teams own and improve them.
AI-written summary · about
chapters (10)
- 0:00Start
- 4:50Incredibly Inconsistent
- 6:30Overhauling the System
- 10:26Make ALL Alerts Actionable a Make Alerts Mutable
- 12:58Tracking Alert History
- 14:16On boarding is a breeze
- 14:57If an alert goes off you have to address it
- 15:51All alerts must be actionable
- 16:47Developers began helping to improve our monitoring system
- 18:26Questions?
related talks
38mDemocratizing Observability
23mMastering Monitoring & Alerting: Key to Seamless Performance
36mThe road to world-class observability
24mWait for Us! Evolving On-Call as Your Company Grows
27mWant to Solve Over-Monitoring...
16mMonitoring your platform from multiple locations
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
36m"Alert overload: How to adopt a microservices architecture..."