You Can't Stop Fires with an Ambulance
summary
Alerts double during cloud migration and incident count keeps rising despite reduced MTTR; 30 SRE engineers build a symptom-first alerting stack, a Klaxon status-callback alert that fires on status page traffic, and a multivet incident chatbot that auto-generates post-mortems and links. Post-mortems reveal release process as the top contributor, only ~50% of remediation work gets completed, and a lightweight report card operational hygiene score drives leadership prioritization and roadmap conversations.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:19Xero - Beautiful Accounting Software
- 4:57So we simplified alerting
- 8:58Invested in incident management
- 10:45Detection and Restoration of Service
- 11:14But the number of incidents is still a problem
- 11:47So I went looking for why
- 12:36Analysis of contributing factors
- 13:36Our Post-Mortems weren't working as expected
- 15:04Finding the influencers
- 15:39Conversation Starters
- 17:05Report Card
related talks
34mIncidents Happen
29mElevating Open-Source Monitoring Ecosystems
45mIncident Management - Talk the Talk, Walk the Walk
52mPersistent SRE Antipatterns: Pitfalls On the...
23mFirefighting: The Engineer's Guide to On-Call
47mPersistent SRE Antipatterns: Pitfalls on the Road to Creating a Successful
23mIncident Response, Management & Alerts - where they fit in CloudOps?
36m"Alert overload: How to adopt a microservices architecture..."