How we improved our monitoring so that everyone likes to be on-call
summary
Team reduces false positives by deleting information-only checks, splits pages into 24/7 and 9–5 hotlines so nights drop sharply, and adds sub-process endpoints to move from single end-to-end checks to actionable metrics. Chaos engineering with Chaos Monkey and outage drills build experience; paging volume is tracked and discussed daily. A rotating “Encore” role handles 15-minute quick wins and handovers; outcome: far fewer wake-ups, bugs found before customers, and higher on-call confidence.
AI-written summary · about
chapters (17)
- 0:00Introduction to on-call monitoring
- 2:27Problems with on-call culture
- 4:12Initial team struggles and burnout
- 5:15Reducing false alarms and alerts
- 6:40Detailed monitoring for sub-processes
- 8:05Training engineers for real outages
- 10:35Testing in production environments
- 12:17Tracking and measuring paging alerts
- 13:45Qualitative feedback from engineers
- 14:28Blameless post-mortems and fixes
- 15:11Progress to active engagement
- 15:33Predictive alarming and its limits
- 16:16Assigning monitoring responsibilities
- 17:18On-call rotation process
- 18:05SLO-based monitoring approach
- 18:48Achieving a happy on-call culture
- 19:35Q&A session
related talks
32mCognitive Apprenticeship in Practice with Alert Triage Hour of Power
06mGo Get Your Developers Involved
45mLet's play a real life Dungeon and Dragons
44mEnhancing the Performance of Engineering Teams
12mConvergent Quality Engineering
09mMaking "developers on support" work for everyone
05m3AM, a Survey
19mLeveraging Analytics for Technical Efficiency and Enhanced User Experience