Antics, Drift, and Chaos
summary
Claim that complex distributed services inevitably drift into failure is illustrated with concrete incidents: a missing functional-test annotation causes a cold-start outage, logging threads holding a Kafka lock throttle CPU and break streaming, and retry storms amplify latency. Mitigations include Chaos Monkey—terminates 5–10 instances per minute during business hours—and ChAP, which clones control and experiment clusters and injects RPC latency while measuring SPS; experiments run in production with auto-stop logic within ~1 minute and automated canary analysis.
AI-written summary · about
chapters (44)
- 0:00Intro
- 0:11Antics, drift and chaos
- 0:33Executes tests to warm
- 0:43Add a new test
- 1:39Result: execution of unit test led to an outage
- 1:49Moral: use unit tests sparingly
- 2:02Complex systems exhibit unexpected behavior
- 3:16Generalized Uncertainty Principle
- 3:32Error handling
- 5:26Scenario
- 5:50Load increases
- 6:51Support systems
- 8:21Non-critical service failed
- 8:48Lock shared by app threads
- 8:59Lock contention
- 9:10Logging took down prod
- 9:20Mitigation
- 9:34S3 billing process was slow
- 9:56Command input entered incorrectly
- 10:37Recap: Antics
- 11:00Act II: Drift
- 12:23Drift into failure
- 12:39Unruly technology
- 15:24Scarcity and competition
- 15:52Efficiency vs thoroughness
- 16:43Decrementalism
- 20:11Sensitive dependence on initial conditions
- 21:57Traffic spike
- 22:43EVcache client defaults to treating errors as cache misses
- 23:20Recap: Drift
- 25:48Chaos engineering
- 26:08Find vulnerabilities before they become outages
- 26:28External validity
- 26:56Risk: vulnerable to failure of non-critical services
- 29:18Build a hypothesis around steady state behavior
- 30:01Vary real-world events
- 30:13Fail RPC calls
- 30:36Run experiments in production
- 30:50Automate experiments to run continuously
- 31:41Minimize blast radius
- 32:11Takeaways
- 32:26Chaos experiments can find pathologies
- 32:44Chaos provides incentives
- 32:54Read these books
related talks
38mChaos Engineering
29mDeprecating Simplicity
32mClear the ring for Chaos Engineering
33mMonkey Business For Serious Systems: Why chaos engineering isnt crazy
29mComplexity: The Crucial Ingredient in Your Kitchen
49mStability Patterns & Antipatterns
01h 04mHarnessing Chaos; the hidden ingredient behind building better systems through learning
03mDeveloping a Chaos Architecture Mindset