You put out the fire on production, what's next?
summary
After a production outage bring systems back fast (example: a 10× marketing spike resolved by scaling out) then wait 2–3 days before analysis to let the team recover. Run postmortems for user-visible downtime, data loss or long on‑call toil per Google SRE guidance. Build a blameless culture, compile a millisecond‑level timeline, use broader RCA like a fishbone diagram (not just Five Whys), create small automated action items with owners and deadlines, publish tagged reports and run simulations to verify fixes.
AI-written summary · about
chapters (17)
- 0:00Introduction and context
- 3:05Key takeaways from postmortems
- 5:11Why failures happen
- 6:13Bringing systems back to life
- 8:42Team recovery after outages
- 9:25When to run a postmortem
- 12:37Documenting postmortems
- 13:00Starting postmortem practices
- 14:47Blameless culture explained
- 17:15Preparing for postmortems
- 18:43Building the incident timeline
- 20:10Root cause analysis methods
- 25:26Defining actionable items
- 28:17Who should attend postmortems
- 32:35Focus on learning over blame
- 36:07After the postmortem
- 40:00Final key takeaways
related talks
23mYou put out the fire on production, what’s next?
22mAfter the dust has settled: learning from incidents
15mMaintaining reliable systems
44mYou build it, you run it
40mProduction Detectives: Solve the Case Before the Crash
24mThe Formula for Faster Outage Recovery
27mManaging Incidents
31mThe Art of Turning Incidents into Opportunities