War Room Warrior: How to manage war room situations
summary
A manhole fire severs a 10 gig link and the network falls back to a 1 gig backup, causing frontend retries to spike firewall load and slow the site; teams use PagerDuty + New Relic + Slack and a Hangouts war room with runbooks and a CAB to triage. The immediate remedy is removed frontend retries and targeted infra fixes, followed by a post‑mortem that audits hidden retries and confirms the ordered 10 gig backup never arrived.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:14About Raj
- 2:05Why is it so critical
- 3:15Its also about revenue
- 3:58What questions do you ask
- 4:48Key metrics
- 5:13Preparations
- 6:22Organization
- 7:17Healthcaregov
- 7:48Monitoring and alerting
- 8:28Mean time to resolution
- 9:23Follow the Sun approach
- 10:07Runbooks
- 10:52Change Management
- 13:49Example
- 14:19External Communication
- 14:54The Real Deal
- 15:55Definitions
- 16:26Incident Managers
- 19:30Boundaries of ownership
- 21:13Slipups
- 22:08Postmortem
- 23:04Sample Postmortem
- 23:33Conclusion
related talks
33mFamiliar Smells I’ve Detected in Your Systems Engineering Organization and How to Fix Them
30mIncident Groundhog Day
27mIt's Down! Simulating Incidents in Production
28mTop 5 Things You Can Do To Reduce Operational Load
30mIt's Down! Simulating Incidents in Production
03mDon't gamble when it comes to reliability
25m"The website's down!" Stories and lessons on keeping your website up.
25mThe story of firefighting: learnings from the incident management