Accept Partial Failures, Minimize Service Loss
summary
Baidu describes a pragmatic strategy that reduces user-impact during failures by isolating faults instead of full recovery. It enforces N+1 data center redundancy, limits automated operations to one cell, and uses local per-cell storage to accept temporary inconsistency. Non-essential subsystems are made detachable so pages degrade gracefully. Traffic triage drops low-value requests and reserves capacity for paid and long queries using real-time query classification with ML. Periodic stress tests and staged rollouts confirm the approach works.
AI-written summary · about
chapters (9)
- 0:00Start
- 2:04Root cause recovery VS partial recovery
- 2:42Basic principles
- 3:22Deployment isolation
- 4:15Deploy isolation - Global Single Point
- 5:37Deploy isolation - Service Across Cells
- 6:07Deploy isolation - Capacity Redundancy
- 10:12Module isolation - External Dependencies
- 11:17User traffic isolation
related talks
36mDesigning for failure - architecting fault-tolerant system
34mDistributed Systems - What Can Go Wrong and How to Get It Right
35mDesign patterns for application resiliency
10mBuilding an 'Oops-Less Operation' at Scale
01h 03mPatterns for Resilient Architecture
29mBuilding Resilient Systems
53mResiliency and Availability Design Patterns for the Cloud
55mGoing Resilient