Expect the Unexpected: Preparing SRE Teams for Responding to Novel Failures
summary
Shopify reports $1,700 USD/sec at stake and an SRE org grown from ~30 to ~120; preparation emphasizes specific practices: open incident channels (Slack bots recording everything), incident simulations for onboarding, controlled game days that induce real production failures, routine exercises to turn rusty knobs, and automated chaos tests in CI. Claim: successful SREs see increasing novelty in failures, so the team builds trust, blamelessness and creativity. Example large exercise runs with ~100 people in an office and 20,000 pieces of Lego.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:30Help About
- 1:48What We'll Cover
- 4:52Transparent Response
- 7:47Game Days
- 10:36Automated Failure Tests
- 15:34Magic Recipe for Novel Failures
- 17:23Training Exercise Formula
- 20:34Summing Up
