Draining the Flood—A Combat against Alert Fatigue
summary
Baidu’s Argus alert stream averages >1.3k alerts per engineer per week (130/day, 100 daytime, 30 night), only ~15% are actionable and ~58% are duplicates. Solutions include linker-buffer grouping (same rule/instance, association rules across modules, network-coverage heuristic), attention-ratio calibration (night ~25%), an escalation tool with staged on-call schedules, and callback self‑healing that removes ~35% of alerts; combined deployment cuts total alerts by ~85%.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:31Problem Statement
- 3:08Observations
- 6:45Grouping Method
- 9:20Grouping Results
- 10:54Grouping Policy
- 12:11Association Rules
- 15:36Network Issues
- 17:14Linker Buffer
- 17:52Attention Ratio
- 19:36Importance Levels
- 22:15Escalation Tool
- 23:12Interface
- 25:17Automatic SelfHealing
- 27:05Management Support
- 27:45Results
- 28:14Conclusion
related talks
16mUnderstanding Alert Fatigue- 16mUnderstanding Alert Fatigue
23mFirefighting: The Engineer's Guide to On-Call
38mEliminating Alerts or “Operation Forest”
33mTackling Alert Fatigue with SLOs
36m"Alert overload: How to adopt a microservices architecture..."
28mAlerts don't suck, YOUR alerts suck
27mWant to Solve Over-Monitoring...