Have You Tried Turning It off and Turning It on Again?
summary
Argues that untested fallback plans and hidden service cycles, not abstract “backend” problems, make global restarts unrecoverable; cites real failures—filesystem-snapshot backups broken by in‑memory metadata, failover sites tripped by timezone replay—and a microservice currency example that creates surprising dependencies. Recommends building a global dependency picture via monitoring, visualizing with Graphviz or Netflix’s Vizceral, enforcing client‑side RPC ACLs and Kubernetes NetworkPolicy, layering systems to be bootstrappable, automating DR tests, and keeping one sealed reserve parachute fallback that’s simple and expert‑maintained.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 6:21Fallback plans dangers
- 15:04Dependency cycles
- 24:18Managing dependencies
- 39:45Key takeaways
related talks
51mDesigning for Failure: Build Resilient Services
16mEngineering Failure-Resilient Systems
22mResilience Is Not Enough
51mWithout Resilience, Nothing Else Matters
23mDressing to the 9s
10mBuilding an 'Oops-Less Operation' at Scale
45mDeveloping a Chaos Architecture Mindset
52mPerformance & Stability Anti patterns