Incident Ready: Chaos Engineer Your Incident Response
summary
Shows concrete experiments: a timed surprise meeting where trying to update Statuspage.io exposes lack of access in 30 seconds, and scripted breaks of staging that are recorded over Zoom while a teammate is asked to roll back a benign production change. Breakdowns reveal missing permissions, absent runbooks, and multiple rollback methods—one unexpectedly faster. Recommends cataloging and practicing small reusable technique bricks (finding logs, purging cache, reverting deploys) and automating boring steps with tools like FireHydrant to remove friction.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:26Who am I?
- 1:06What are we talking about?
- 1:28Quick overview of chaos engineering
- 2:14Examples of chaos experiments
- 2:50Process Experiments
- 3:39So how do you run an experiment on process?
- 3:49Surprise Meeting + What we learned
- 4:53Do this for any common operation you do during an incident
- 5:32People mitigate incidents, not processes
- 5:59Guide, don't prescribe
- 6:50Why this matters
- 7:24Rearrange blocks for different scenarios
- 8:09The bricks are boring
- 8:45Example: marching band technique
- 9:32Identify which techniques lack undestanding
- 13:15Then practice them
- 14:15What did we cover?
- 14:47Thank you!
related talks
22mBe prepared for production incidents
30mHow To Have An Operational Incident (a crash course)
29mComposing the Response: Building an Incident Pipeline from Scratch
31mThe Art of Turning Incidents into Opportunities
44mPanel: Peak Performance Under Pressure: Building Cross-Functional Resilience in Incident Response
48mChaos Engineering Panel
57mThe Curious Incident in your Software in the Day Time
28mRevolutionizing Incident Response