The Hidden Costs of On-Call: False Alarms
summary
False alerts multiply costs: simple investigation time plus a ~40% context‑switch penalty produces roughly a 4× overhead and a 5% false‑alert rate can cost tens of thousands annually (author’s example pulls ~$39,000 from the couch). Practical fixes include 182→8–12 pagers/week after tuning, use of composite monitors, prefer user‑experience metrics over host checks, route informational messages to syslog/Splunk not inbox, and automate safe run‑book repairs while avoiding destructive global restarts.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:54Hidden Costs
- 2:27Investigation
- 7:58Cognitive Load
- 9:23Stress
- 12:46Sleep Deprivation
- 14:52Work-Life Balance
- 17:15Alarm Fatigue
- 21:12Check the Right Things
- 22:11Limit Reminders & Combine Related Alerts
- 26:52Fix System Fragilities
- 28:33Your monitoring system is a constant reminder of work not put into it
related talks
19mSensory Friendly Monitoring: Keeping the Noise Down
19mIts Time We Rethink On-Call Culture
23mFirefighting: The Engineer's Guide to On-Call
19mSensory Friendly Monitoring–Keep the Noise Down
30mOn call does not have to suck
16mUnderstanding Alert Fatigue
27mWant to Solve Over-Monitoring...
06mIgnite: Key Factors for Successful 24×7 On-Call Duty