From Hammer to Scalpel
summary
Uber holds a 10-minute SLA to mitigate user-facing outages and runs 4,000 microservices with ~300 deploys/hour and 1,500 flag changes/hour, so root cause can take hours. The emergency path uses a global marketplace failover via a CLI that updates city placement and issues 307 redirects to another region, mitigating in minutes at cost to productivity. Newer zonal Muxy drain scalpel removes specific service zones in seconds. A recent Redis bug took 4+ hours to root cause but mitigation saved $4M.
AI-written summary · about
chapters (10)
- 0:00FROM HAMMER TO SCALPEL: UBER'S CHANGING MITIGATION STRATEGIES FOR MAJOR USER FACING OUTAGES
- 4:40Multiple Deployment Approaches
- 9:11The Swiss Cheese Model of Reliability Hazards
- 9:58The Challenges From Infrastructure Complexity
- 13:26The Bad News
- 18:35Big Hammer Costs
- 21:05Saved By the Hammer
- 21:56The Underlying Technology Behind The Scalpel
- 27:26The Best of Both Worlds
- 33:33General Lessons for Effective Outage Management
related talks
18mMaking what you have enough
27mLessons Learned From Architecting the Highest-scale Operational Systems in the World
28mFixing On-Call When Nobody Thinks It's (Too) Broken
30mLearning from Learnings: Anatomy of Three Incidents
30mLearning From Failure: Post Mortems
35mSCALA BASED SERVICES RESILIENCE IN THE REAL WORLD
25mSRE and adapting to the uncertainties of military conflict
43mFrom Developer to Development Manager