Meeting Impossible SLAs: How we made our uptime 99.999%
summary
Claims a 99.999% uptime SLA and demonstrates it holds during a recent US‑East‑1 DynamoDB/DNS outage by combining math, tooling, and procedures. Architecture runs in AWS with CloudFront edge compute, Route53 DNS failover driven by active health checks, six primary + six backup regions, and a retry policy that proves any dependency below 99.7% cutoff is unusable (90% component needs five retries but retry handler limits to two). Adds prod validation tests, anomaly detection on authorization ratio, WAF + 2000 RPS rate limits and JA3/JA4 fingerprinting; the system stays up.
AI-written summary · about
chapters (15)
- 0:00Impossible uptime SLAs
- 1:40999% uptime requirements
- 3:27Failure modes in basic architectures
- 5:56Reliability of third-party components
- 9:31Regional infrastructure failures
- 13:26Edge compute for resilience
- 14:51Application bugs and rollouts
- 18:42AI for anomaly detection
- 20:46Customer vs system incident views
- 22:31Customer support as uptime lifeline
- 24:57Defending against attacks
- 27:49AWS WAF rate limiting
- 31:46Architecture review for 59s uptime
- 32:49Lessons from failures at scale
- 34:37Conclusion and Q&A
related talks
46mWhat You'll Miss on AWS & How To Find It
30mZero Down Time Operations: How to Get Atleast 3 9's of Uptime
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
30mHow are we transforming the org to achieve 99.99% uptime at DigiCert
47mUptime == Money: High Availability
56mGetting all the 99.99(9) you always wanted
19mPreparing for disaster: going Multi-Region
51mBuilding for Resiliency and Scale in the Cloud