Evolution of Incident Management at Slack
summary
Slack handles over 150 million messages per minute and now runs about a dozen controlled deployments per day using dog food and canary phases with phased rollouts; median incident lasts 124 minutes and two‑thirds resolve under four hours. Incident management trains everyone with a three‑hour responder class, has ~25% of engineers trained as incident commanders, a companywide major IC rotation plus layered slack IC backups, an incident bot and commander checklist, follow‑the‑sun staffing (SF, Melbourne, Dublin, Pune), and uses area command for resource contention.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:26History of Slack
- 2:34Reliability Crisis
- 6:03Incident Management Vision
- 8:00Incident Management Plan
- 10:53Incident Management Training
- 11:56Severity Levels
- 12:53Major IC
- 13:58Major IC oncall
- 15:54Major IC responsibility
- 18:35Simultaneous incidents
- 19:45Area Command
- 21:04Long Duration Incidents
- 22:45Pillar Incidents
- 26:01Whats Next
- 26:41Ongoing Challenges
- 27:12Recruitment and Training
- 27:40Challenge of Success
related talks
19mBalancing fast-paced innovation and technical debt
19mScaling Slack
44mPanel: Peak Performance Under Pressure: Building Cross-Functional Resilience in Incident Response
28mHow we do developer productivity at Slack
42mMoving Fast and Securing Things
32mSustainable Incident Management
04mHighlights - Future of Work
18mStaying in your lane: incident response for leaders