Learning from Incidents at Scale; Actually Doing Cross-Incident Analysis
summary
A four-person resiliency team runs blame-aware reviews for every major and sensitive incident, feeds an incident response bot into Jira incident reports, and fields monthly, quarterly and yearly analyses that break incidents by quarter product and contributing factor. Cross-incident findings trigger concrete projects: rearchitecting a critical customer flow, a quarterly SLO initiative after repeated alerting action items, and a recommendations vs commitments workflow for ownership. The result is broader stakeholder buy-in and measurable prioritization instead of an action-item graveyard.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:05Beyond root causes
- 8:23Scaling incident analysis
- 13:18Innova success story
- 22:31Analysis artifacts
- 25:41Cross-incident analysis
- 27:47Action items trap
- 31:19MTT metrics context
- 33:48Metrics to insights
- 35:09Scaling incident learning
- 36:53Q&A
related talks
37mIncidents are a new normal
34mIncidents are new normal
30mHow Did Things Go Right? Learning More from Incidents
19mIncident Archaeology: Extracting Value from Paperwork and Narratives
19mHow to handle incidents at scale
44mProduction Incident Evaluation as Cost of Quality
20mData Incidents - When things go wrong
39mNavigating the Chaos: A Holistic Approach to Incident Management