Building Ultra-Resilient Systems in Go
summary
A total power loss in the primary data center causes one outage that takes days to recover and five months later the same failure is a non-event; the team moves to active‑active clusters across US and Europe with Consul-style service discovery and leader election for stateful work, splits email into priority topics using the bulkhead pattern, adds timeouts, retries with exponential backoff and client-side circuit breakers, defines SLOs, runs weekly chaos tests that literally pull the plug, and ships Go libraries for these patterns.
AI-written summary · about
chapters (17)
- 0:00Introduction and incident story
- 3:52CAP theorem trade-offs
- 4:54Principles for resilient systems
- 5:37High availability patterns
- 7:43Active-active deployment in Go
- 9:06Active-passive with leader election
- 11:11Failure isolation and prioritization
- 12:35Bulkhead pattern for workloads
- 14:00CQRS for fault tolerance
- 15:43Handling external dependencies
- 16:24Timeouts and retries in Go
- 18:09Caching for resilience
- 19:55Circuit breaker pattern
- 22:21Service level objectives
- 23:47Chaos testing and validation
- 24:53Real-world outage comparison
- 25:56Key lessons learned
related talks
29mBuilding Resilient Systems
53mResiliency and Availability Design Patterns for the Cloud
55mGoing Resilient
10mBuilding an 'Oops-Less Operation' at Scale
01h 03mPatterns for Resilient Architecture
42mDesigning for failure
35mDesigning for failure
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation