Fixing Your Noisy Pager in 500 Easy Steps
summary
Pager load drops from hundreds of pages per month to low tens per week by combining a weekly triage loop — group alerts by name, sort by frequency, manually mark alerts as mostly correct or noise — with a Kubernetes automation called Auto Repair that reads Prometheus alerts and runs small Go fixes (pod restarts, deletes). Automation has three safety limits: three automated limits (don’t auto-fix if frequent, don’t repeatedly restart same instance, don’t clear evidence of a fired page). Results: meaningful reduction and ongoing habit-driven improvement.
AI-written summary · about
chapters (7)
- 0:00Identifying the problem
- 1:23Analyzing alert data
- 3:58Categorizing pager alerts
- 8:32Optimizing alert systems
- 10:31Managing external dependencies
- 12:44Implementing auto-repair
- 17:41Long-term results and takeaways
related talks
32mCognitive Apprenticeship in Practice with Alert Triage Hour of Power
53mYour Attention, Please: Optimizing Metrics for Humans
28mThink Backstage! Beyond Plugins
22mHow we improved our monitoring so that everyone likes to be on-call
48muses PagerDuty to speed up DevOps & cloud innovation
36mDeveloper Productivity On a Budget
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
21mSensory Friendly Monitoring: Keeping Noise Down