How to stay alive even when others go down:...
summary
Shopify runs about 300k requests per minute and finds strict client timeouts cut cascading failures; circuit breakers stop repeated retries and are implemented at the driver level; a test-time proxy called taxi proxy simulates real connection failures so integration tests exercise error paths; a semaphore system (Semyon) throttles DB concurrency to prevent overload; a resiliency matrix turns unknowns red into concrete fixes; chaos testing starts manual (kill nodes, fill disks) and moves toward automation.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:38Goal
- 1:30Whos others
- 2:19What is service
- 3:17Examples of services
- 3:58Goals
- 5:23Capacity
- 9:19Failure
- 10:04Ruby
- 14:02Circuit Breaker
- 15:19How it works
- 16:43Implementation tip
- 18:12Fading gracefully
- 19:21Example
- 21:10Examples
- 22:44Monitoring and alerting
- 25:27BackHeading
- 27:52Database overload
- 29:37Where to start
- 31:26Resiliency matrix
- 32:23Resiliency testing
- 33:28Module overview
- 35:44Taxi Proxy
- 37:19Cares Monkey
- 40:19Resources
related talks
28mBuilding and testing resilient applications
33mResilient
32mSurviving Black Friday - A resilience engineering tale
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
42mPlanning To Fail
52mPerformance & Stability Anti patterns
35mDesign patterns for application resiliency
16mEngineering Failure-Resilient Systems