The Inevitable Tech Incident: The Lessons We Just Can't Seem to Learn
summary
Argues that outages are inevitable and that learning fails when lessons stay at the individual level instead of becoming team or organizational practice. Recommends define incident boundaries (what counts as an incident, when to escalate), assign an incident owner with clear responsibilities before outages, build monitoring and alerts proactively with thresholds that tolerate some false positives, document audience-specific incident reports and instrument ownership between frontend/backend, and run dedicated monitoring days or teams; these practices aim to reduce repeat failures.
AI-written summary · about
chapters (7)
- 0:00Start
- 4:32Chaining questions The endless interrogation of reality
- 10:35Creating unnecessary complexity
- 14:13Incident report "audience" Q: Are we writing reports with the audience in mind?
- 15:33Incident report "understanding" Q: How do we make sure the reads understand them?
- 17:36Monitoring Q: Do we have the right monitoring and alerting tools?
- 20:39Higher-level system design Q: Are we abstracting functionality by the skill required?
related talks
37mIncidents are a new normal
15mMaintaining reliable systems
25mPragmatic Incident Response: 5 lessons learned from failures
13mThe Incident Is The Way: Using Your Incidents to Win...
01h 03mWhispers in the Chaos: Monitoring Weak Signals
15mPragmatic Incident Response: Lessons Learned from Failures
19mHow to handle incidents at scale
30mHow Did Things Go Right? Learning More from Incidents