Learnings from Chaos experiments
summary
Summary of concrete learnings from running chaos experiments for ~4–5 years across Fanatics and banks: common failures are reconnect failures, timeout and crash modes, master/slave elections and split‑brain, autoscaling misses, and missed exception handling; tools actually used include Chaos Monkey, Chaos Blade, Pumbaa, Chaos Mesh, Litmus, Chaos Toolkit and VMware Mangle; experiments run with a small blast radius and one failure at a time; fixes go into prioritized backlog, observability and monitors must catch injected faults, and auto‑recovery and error‑budget recording need verification.
AI-written summary · about
