The tools and techniques behind resilient systems
summary
A customer fleet outage hits 2,000 instances after a deployment and a startup script overwhelms an internal API, creating a retry storm; engineers show a client-side adaptive retries approach using a token‑bucket (rate increase/decrease like TCP) that avoids prolonged metastable failures. A demo splits a 100 KB DynamoDB item into six chunks and uses erasure coding (reconstruct from first four), giving ~2378 ms for one GET vs under 1s with erasure coding at ~1.5× storage/bandwidth cost. Backoff helps closed systems but adds little for open public APIs.
AI-written summary · about
chapters (8)
- 0:00Introduction to resilient systems
- 2:12Understanding retry storms
- 5:03Adaptive retries explained
- 9:33Backoff and jitter in systems
- 11:22Metastability and failures
- 14:16Tail latency challenges
- 17:10Erasure coding demo
- 24:57Key takeaways summary
related talks
48mResiliency and Availability Design Patterns for the Cloud
16mEngineering Failure-Resilient Systems
40mThe Art of Embracing Failure at Scale
10mDesigning for Failure using AWS: Resilient Systems
34mToward continuous resilience
18mScaling APIs to Billions: Extreme-Scale Reliability
30mThe last in the line: a talk about DevOps and Disasters
26mA proactive approach to resilience in modern apps