How We Went From Two Major Outages to 99.98% Reliability in 6 Months
summary
Company pauses product engineering for a quarter and focuses on reliability, then hits 99.98% to 99.99% within months by practical changes. Stack uses Kubernetes on Google Cloud with Terraform, ELK APM, Prometheus/Grafana and CircleCI. Engineering splits a Django monolith into eight deployments, adds read replicas and read‑only mode, implements active‑active multi‑region clusters with global load balancer, extends JWT expirations for offline client access, automates on‑call and postmortems, drills scenarios quarterly, and sustains zero churn while surviving 20x traffic.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:40Two major outages crisis
- 3:04Reliability for Twingate
- 5:31Baseline practices
- 5:51Reliability as core feature
- 7:40Reducing blast radius
- 8:25Splitting monolith
- 10:12Read replicas added
- 11:17Disaster drills
- 12:21Non-critical writes handling
- 13:03Automatic read-only mode
- 13:48Multi-region active-active
- 15:33Multi-region benefits
- 16:18Thundering herd problem
- 17:20JWT token expiration
- 18:46Automating on-call
- 20:14Quarterly disaster drills
- 21:19Results and customer impact
- 22:4320x traffic spike handling
- 23:06Reliability benchmarks exceeded
- 23:51Reliability task force plans
- 24:57Sharding production
- 26:00Q&A
related talks
56mGetting all the 99.99(9) you always wanted
42mArchitecting for Reliability
40mStrategies for Zero Down Time, Frequent Deployments
48mDevops@Scale
30mZero Down Time Operations: How to Get Atleast 3 9's of Uptime
45mEngineering resiliency - how I learned to love an outage
55mDevOps @Scale (Greek Tragedy in 3 Acts)
39mOops. I broke the Google, now what?