Alerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
summary
Company runs ~100 engineers with a ~5 person production engineering team and replaces noisy host checks with symptom-based alerts so pages reflect user impact not internal failures. Prometheus plus a blackbox exporter and time‑series trend rules replace hundreds of binary alerts, e.g. pipeline delay replaces many task flaps, yielding the company’s most reliable quarter. Use whitebox for imminent problems, silence noisy host-level checks, convert repeat runbook steps into automated responses, and keep runbooks as scripts for on-call diagnostics.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 3:32SRE by the Book
- 5:16Scalability
- 6:47The 50 Law
- 10:20Monitoring Alerts
- 14:33Symptoms vs Causes
- 15:37Why is this new
- 16:59Blackbox vs Whitebox
- 18:02Pros and Cons
- 20:07Impending Problems
- 23:57SymptomBased Monitoring
- 24:48Nagios
- 26:37Anomalies
- 27:26Silence
- 28:17Runbooks
- 30:03Conclusion
related talks
18mAlert Analytics
27mWant to Solve Over-Monitoring...
25mSymptom-based Alerting for Machine Learning - What I Learned...
50mApplied Alerting Philosophy
33mAlerting in Cloud Native Environments
46mCluster Golden Signals to Avoid Alert Fatigue at Scale
27mBuilding a Scalable Monitoring System
19mSensory Friendly Monitoring: Keeping the Noise Down