Failure Happens: Improving Incident Response in Large-Scale Organizations
summary
Describes concrete fixes for slow enterprise incident response: turn manual, ticket-driven handoffs into operations as a service with a secure hub, procedures-as-code, and audited self-service (Rundeck-style orchestration). Uses a real 13-hour outage that ballooned a context wagon into dozens of people to justify shifting decisions closer to the keyboard. Shows measurable results — e.g. Ticketmaster cuts mean time to restore from 47 minutes to 3.8 minutes and halves escalations — and outlines runbooks, SLAs and governance to prevent heroics.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:20Case Study
- 2:58The Problem
- 7:40Business Approval
- 11:06DevOps
- 15:39Mean Time to Detect
- 20:58Shift Left
- 27:06Operations as a Service
- 29:05Automated Procedures
- 34:41Think First
- 36:04Ticketmaster
- 40:52Recap
related talks
30mKeeping Your DevOps Transformation from Crushing Your Ops Capacity
05mDevOps and incident management: A recipe for success
31mFuturistic Luxury Incident Response
17mIncident Response @ Scale-Building a Next Generation SOC
27mManaging Incidents
30mHow To Have An Operational Incident (a crash course)
31mThe Art of Turning Incidents into Opportunities
34mIncidents Happen