Incidents are a new normal
summary
Argues incidents are inevitable but preventable from repeating by running blameless post-incident reviews, using a facilitator to stitch chat logs, dashboards and a Dynatrace-generated timeline, and by applying five whys to move from causal factors to root causes. Gives concrete examples: a rollback that worsened outages, a locked database from untested write load, and fixes that worked (feature flags, canaries, added monitoring). Requires actionable items with owners, deadlines, and one mandatory step: share learnings across teams.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:14Presentation
- 10:43Biases
- 15:05Blameless
- 18:34Timeline
- 19:13Timeline isnt linear
- 21:08Questions
- 22:45We are ready to fail
- 23:33Causal factors
- 24:35Root causes
- 25:44Example
- 26:31Myths
- 28:21Minimize Consequences
- 30:08Good Action
- 34:43Mandatory actions
- 35:27Summary
related talks
34mIncidents are new normal
28mThe Inevitable Tech Incident: The Lessons We Just Can't Seem to Learn
13mThe Incident Is The Way: Using Your Incidents to Win...
40mBuilding Resilience: How to Learn More from Incidents
37mLearning from Incidents at Scale; Actually Doing Cross-Incident Analysis
22mAfter the dust has settled: learning from incidents
20mData Incidents - When things go wrong
19mHow to handle incidents at scale