A Retrospective: Five Years Later, Was Chaos Engineering Worth It?
summary
Five years of practice yields concrete verdicts: chaos experiments are cheap, automatable regression checks that catch real bugs. Examples include kubelet binds to first API server (causing node recovery failures), Alpine musl lacks DNS over TCP (DNS truncation breaks at ~48 nodes), and deliberately introducing 15 chaos points inside a certificate-rotation service to expose implementation bugs. Tools used include PowerfulSeal, Litmus, Pumba and Toxiproxy; upgrades and chaos reliably reproduce subtle golang/kubernetes issues and deliver ROI from confidence to critical bug discovery.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:00What is Chaos Engineering
- 3:17The Innovation Adoption Lifecycle
- 4:38The Mobile Phone
- 6:15The Chaos Monkey
- 8:50Examples from the trenches
- 9:46Are you really highly available
- 14:26Resolving DNS
- 14:53Truncation
- 18:49Finding Bugs
- 20:36Causes of Failure
- 21:49Lessons Learned
- 26:11Tools
- 28:54Training
related talks
36mChaos Engineering in 2021
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure
38mChaos Engineering
05mLightning Talk: Chaos Engineering In Practice
48mChaos Engineering: The Path to Reliability
39mChaos: Breaking your Systems to Make them Unbreakable
29mBoF Deep Dive: Chaos Engineering
19mLearnings from Chaos experiments