The Anatomy of a Cascading Failure
summary
Shows how a rollout bug triggered a RDS Postgres spike of unexpected writes, how AWS burst limits throttled the database, and how a cron job every 15 minutes produced an asynchronous DDoS that blew up downstream services. Details fixes: moved from brittle SDKs to an Envoy sidecar model, added idempotent endpoints, jittered exponential backoff, a 10% retry-budget heuristic, gRPC timeout propagation, Kafka-style publish/subscribe buffering, and chaos tests; these changes reduce retry storms and improve observability and recovery.
AI-written summary · about
chapters (15)
- 0:00Introduction to 26 Bank
- 1:04What is a cascading failure?
- 2:28Triggers of cascading failures
- 6:44Resource interdependence in failures
- 8:10How failures propagate
- 11:43Mitigation strategies overview
- 12:53Capacity planning alternatives
- 15:22Chaos testing introduction
- 16:03Idempotency and statelessness
- 17:09Retry strategies and jitter
- 18:41Timeout discipline and propagation
- 21:11Rate limiting and circuit breaking
- 24:23Fallback strategies for failures
- 25:50Tooling for resilience features
- 29:44Observability and final advice
related talks
42mArchitecting for Failures in Microservices
54mAdrian’s Greatest Hits, B-Sides & Re-issues
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)
20mScaling Microservices and Distributed Databases in a Multi-Cloud World
51mDesigning for Failure: Build Resilient Services
43mDesigning and building resilient backend APIs in the cloud
28mNavigating the Transition: SRE Challenges and Highlights
35mSCALA BASED SERVICES RESILIENCE IN THE REAL WORLD