Watering the Roots of Resilience: Learning from Failure with Decision Trees
summary
Argues that aligning mental models with reality requires chaos experiments plus documented decision trees; shows an open‑source YAML tool (deciduous) to generate trees, use cases like a replicated DB outage where severing replication revealed tighter coupling, a design change to halt the secondary, and a rerun that surprisingly caught up faster; gives a real result where a firewall only blocked the change 60% of the time and a config tool filled the gap; recommends smoke tests, monitoring and iterative experiments in staging.
AI-written summary · about
chapters (13)
- 0:00Resilience and mental models
- 1:10Adaptation in complex systems
- 6:03Resilience potion ingredients
- 10:17Chaos experiments for systems
- 13:26Chaos vs other tests
- 17:15Security chaos engineering
- 19:20Security chaos experiments
- 22:27Capturing knowledge with trees
- 23:50Visualizing system resilience
- 31:32Decision trees in SRE
- 33:56Trees in incident reviews
- 37:28Refining mental models
- 39:15Summary
related talks
29mLooking under the rug: the art of learning from failure
16mEngineering Failure-Resilient Systems
19mFireside Chat with Jason Yee
43mResilience Engineering Mythbusting
49mDeveloping a Culture of Reliability & Resilience through Chaos
44mDeveloping a Culture of Reliability and Resilience through Chaos
30mYou've Lost That Process Feeling: Some Lessons from Resilience Engineering
48mChaos Engineering: The Path to Reliability