Shrinking the Impact of Production Incidents
summary
Argues that postmortems are the primary input for prioritizing reliability work and shows how to cut user pain by improving time to detect, time to mitigate, and time between failures. Recommends concrete tools: SLIs/SLOs tied to client measurements, actionable alerts, runbooks, dashboards, automated rollbacks, weekly failure drills and chaos tests. Notes responders handle ~two incidents per shift and each extra “nine” of availability roughly multiplies cost by ten, so tune targets to user needs.
AI-written summary · about
chapters (8)
- 0:00Production incident impact
- 3:39Outage phases and user happiness
- 6:48Learning from incidents
- 9:57Blameless incident response
- 11:41Faster incident detection
- 17:41Faster incident mitigation
- 25:50Engineering for fewer outages
- 28:16Key takeaways
related talks
51mShrinking Production Incidents
15mMaintaining reliable systems
15mPragmatic Incident Response: Lessons Learned from Failures
25mHow to detangle and understand production
47mCultivating Production Excellence
36mSurviving in Production: Incident Management and Learning
23mYou put out the fire on production, what’s next?
29mReducing Trauma in Production