Distributed Systems - What Can Go Wrong and How to Get It Right
summary
Shows concrete failure modes and fixes: a 1M→2M request spike plus naive 3-retry clients multiply traffic (4x) and cause the spiral of death; use a token bucket (0.01 per success → ~1% retry) to limit retries. Demonstrates cache fallback risk with Postgres/OpenAI-style overload, recommends hardening primaries not blind fallbacks. Quantifies limits (about 150 successful requests before latency hits 200ms / total collapse near 240) and promotes load shedding and constant work (bounded, predictable updates) as practical remedies.
AI-written summary · about
chapters (7)
- 0:00Introduction to failures
- 1:44The dangers of naive retries
- 8:31Understanding fallback patterns
- 12:50Improving system reliability
- 15:50Load shedding techniques
- 22:45Designing for constant work
- 27:18Summary and Q&A
related talks
29mHorrors of Distributed Systems
25mAvoiding Overload in Distributed Systems
01h 01mLittle things and big things: lessons for building large systems.
32mHow does your application behaves when everything goes wrong?
36mDesigning for failure - architecting fault-tolerant system
53mResiliency and Availability Design Patterns for the Cloud
56mMastering Cloud Scalability and System Designs Deep dive
31mWhat Breaks Our Systems: A Taxonomy of Black Swans