The Multi Layered Cake of Resilience
summary
Shopify handles peaks like 76 million requests per minute and 14 million MySQL queries per second and builds resilience by accepting failure at every layer. Practical steps: set 60 second timeouts at the nginx edge but much shorter outgoing HTTP and app timeouts; enforce two second DB max execution (or PT-kill for long queries); add Simeon circuit breakers around HTTP, MySQL, Redis and use capacity control (checkout throttles, client rate limits) to shed load before cascading failures.
AI-written summary · about
chapters (6)
- 0:00Multi-layered resilience intro
- 3:51System reliability basics
- 7:44Timeouts and circuit breakers
- 10:11Simulating slow query deg
- 13:20DB slowdown cascading fails
- 17:09Capacity control and load shed
related talks
32mHow does your application behaves when everything goes wrong?
34mDistributed Systems - What Can Go Wrong and How to Get It Right
41mCapacity and Stability Patterns
53mResiliency and Availability Design Patterns for the Cloud
15mBoosting System Resilience: Exponential Backoff
35mDesign patterns for application resiliency
01h 03mPatterns for Resilient Architecture
23mHow Flash Sales Improved Our Capacity to Handle Scale