Blast Radius Reduction for Large-Scale Distributed Systems
summary
Argues that splitting a global cluster into ~100+ cell-based architecture units limits spatial blast radius but raises infra cost (example: one 100-cell design needs ~5GB per cell vs a single pool) and ops overhead solved by infrastructure-as-code and automation; shows multi-region cell replicas for HA, a control-plane cell router/controller mapping tenants to cells, and a deployed DNS-based global load balancing (plus L4/L7 and BGP options) where DNS TTL and BGP flapping cause real failures; explores formal verification with model checking and SMT solvers to prove isolation properties.
AI-written summary · about
chapters (16)
- 0:00Blast radius reduction intro
- 1:00Large-scale system complexity
- 2:28Blast radius defined
- 3:10Balancing blast radius, elasticity, cost
- 4:38Layered isolation strategies
- 5:20Cell-based architecture basics
- 6:23Data sync in cell-based systems
- 7:30Cell-based DNS isolation
- 8:38High availability across cells
- 9:19Single-cell microservice design
- 10:21Cell-based costs and challenges
- 11:26Temporal recovery strategies
- 12:51Global load balancing
- 13:58DNS vs BGP traffic steering
- 14:19Lamport exposure: client reliability
- 15:01Formal methods for reliability
related talks
40mThe Art of Embracing Failure at Scale
51mPatterns for building resilient software systems
31mThe Myth of the Monocluster
01h 01mLittle things and big things: lessons for building large systems.
01h 03mPatterns for Resilient Architecture
18mScaling APIs to Billions: Extreme-Scale Reliability
01h 02mTools for simplified large scale systems
36mDesigning for failure - architecting fault-tolerant system