Next-generation Alerting and Fault Detection
summary
Machine learning and stream processing often underdeliver for alerting: commercial attempts (Etsy Skyline) and large-scale ops favor simple, maintainable rules. Practical approach uses an alerting IDE (bosun) with tag-based queries, the band function to compare current median to historical medians, and an erraticness factor — a ratio of current window stddev to historical stddev — to catch sudden drops (values ~6–9 signal trouble; example hit 9.4). Bosun adds dependency logic, macros, historical simulation and supports graphite, OpenTSDB, InfluxDB and Elasticsearch.
AI-written summary · about
chapters (17)
- 0:00Start
- 2:20Signal to Noise Ratio
- 2:56Static Threshold
- 5:29Machine Learning
- 6:36What Is an Anomaly
- 9:06Signal Strength
- 9:24Security Logs
- 12:10Matrix Metadata
- 18:29Operational Complexity
- 21:48Rule Editor Interface
- 23:23Key Features
- 25:22Dependencies
- 26:01Transient Time Series
- 26:39Macros
- 31:37Scenarios in Which It Could Look Bad
- 34:02The Threshold of the Standard Deviation
- 37:55Infrastructure Complexity
related talks
50mApplied Alerting Philosophy
25mSymptom-based Alerting for Machine Learning - What I Learned...
38mWatch Everything, Watch Anything: Anomaly Detection
33mAlerting in Cloud Native Environments
32mWho Are Your Alerts For?
41mKnow your data: The stats behind your alerts
54mBosun: modern monitoring
37mSmart Data-driven Alerts (SDdA) and How They Make DevOps Life Easier