Automating Chaos Attacks at Expedia
summary
Expedia describes running CI resilience tests that use testcontainers and toxiproxy to cut the connection to a ML sorting service and verify an explicit fallback to date-sorted reviews, plus automated production attacks that randomly terminate Kubernetes pods with CubeMonkey. A single pod kill reveals cascading failures from poor health checks and misconfigured autoscaling, and new replicas show cold-start latency that the team mitigates with an open source maintenance warmup. Tests integrate with dashboards, alerts and opt-in annotations so teams own hypotheses and fixes.
AI-written summary · about
chapters (5)
- 0:00Introduction to Chaos Engineering
- 1:10Fault injection in applications
- 8:22Platform attacks on Kubernetes
- 17:59Chaos engineering misconceptions
- 19:49Q&A session
related talks
39mResilience & Chaos Engineering in the Cloud
19mChaos Engineering at Scale
40mJourney into the World of Chaos Engineering
46mDesigning a System for Chaos
46mChaos Experiments - Datacenter Stress Testing
37mCreating chaos in containers
57mChaos Engineering Primer
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure