Are We Getting Better Yet? Progress Toward Safer Operations
summary
Argues that counting incidents or MTTR obscures progress and pushes for a shift from prevent-and-fix to a learn-and-adapt approach, showing concrete artifacts: Jelly for Slack analysis, Zoom transcripts as artifacts, interview debriefs, annotated timelines and contributing-factor tags. Demonstrates overlaying diagnosis and mitigation into a combined diagnostic+mitigation timeline, using SLOs and meta-analysis across incidents to surface hidden influences, and shows these practices uncover operational surprises, coordination gaps, and tacit knowledge that narrow dashboards and single metrics miss.
AI-written summary · about
chapters (22)
- 0:00Complexity
- 2:42Law of Stretched Systems
- 5:04Prioritize a learn and adapt safety mode over a prevent and fix safety mode
- 5:19Prevent & Fix
- 5:50Learn & Adapt
- 8:09Measuring progress
- 11:00Metrics anchor the story and the story gives meaning to the metrics
- 12:01Barriers and guardrails are used to prevent people from repeating mistakes
- 13:28Performance variability
- 13:39Ensure positive outcomes through activities like team practice and chaos experiments
- 14:37Chaos experiments as scrimmage
- 19:07Incidents are a source of insights
- 20:05Service Level Objectives
- 20:35Control vs Influence
- 22:12Watch the inputs Influence the outputs
- 23:14Opportunity vs Obligation
- 32:24Judging human performance with metrics applies conclusions without context
- 33:26Recording performance metrics promotes one perspective over others
- 35:15Interview Debriefing
- 39:08Ask deeper questions
- 39:31How close to the safety boundary is the pod autoscaler pushing my infrastructure?
- 39:42Recap
related talks
37mIncident Management Metrics That Matter
30mHow Did Things Go Right? Learning More from Incidents
37mIncidents are a new normal
33mThe Problem with MTTR: Learning from Incident Reports
34mIncidents are new normal
24mThe Un-Incident: Extracting Value from the Gray Area of...
29mShrinking the Impact of Production Incidents
01h 04mIncident Management - Talk the Talk, Walk the Walk