Getting all the 99.99(9) you always wanted
summary
Runs production for 70,000 merchants and ~6 million customers and argues uptime depends on three concrete practices: eliminate single points of failure, reliable rollout, detect failures fast. Uses Docker, Kubernetes, AWS and Java, plus Splunk/Zipkin-style tracing, Graphite/Grafana, Kafka/SNS for queues. Deploys a 25% canary group to catch production-only edge cases, builds internal tools (a Zipkin-like tracer and monks health-checks), and shifts monitoring to percentiles after seeing mean 4.3s hide >10% requests >10s. Results: canary + automation + monitoring prevents many outages.
AI-written summary · about
chapters (15)
- 0:00Start
- 22:14Richardson maturity model
- 23:11Service registry
- 25:32Request/Acknowledge
- 28:35Various Service patterns
- 39:33Monks - Internal health checks
- 42:43External checks are the real checks
- 43:05Time series event monitoring
- 45:51Problem 1: Seasonal changes
- 47:05Problem 2: Looking at average
- 47:48Problem 2: Looking at median
- 48:15Problem 2: Another distribution
- 48:57Percentiles for the win
- 50:04Future of time series monitoring
- 51:05Summary
related talks
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
58mCultivating Production Excellence
46mManaging a Billion Transactions with Tech, Tactics, and Teamwork
30mZero Down Time Operations: How to Get Atleast 3 9's of Uptime
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
32mReliability in Finance: Delivering Quality Market Data at Scale
47mUptime == Money: High Availability
23mMastering Monitoring & Alerting: Key to Seamless Performance