Zero Down Time Operations: How to Get Atleast 3 9's of Uptime
summary
37Signals claims consistent three nines (99.9%) uptime by doing concrete things not platitudes: they survive a 197–200 day kernel hang by moving critical services to RHEL/CentOS, replace fragile switch loops with MLAG and blade chassis, and buy capacity. They build tools that work in production: intermission to pause traffic, mysql_role_swap to switch masters in seconds, dashboards and alerts on statsd + nagios, and zero downtime migrations with pt-online-schema-change, which together cut frequent pages and prevented an estimated 10–20 outages.
AI-written summary · about
chapters (36)
- 0:00Intro
- 0:50Background
- 2:18Status Site
- 2:50BaseCamp
- 3:28Scheduled Maintenance
- 3:43Why This Matters
- 4:07The Fortune 500
- 4:33Quiz
- 5:14Definitions
- 5:36Configuration
- 6:18Whats Significant
- 7:00Bugs will rob you of uptime
- 7:44Network Architecture
- 9:22Virtual Link Trunking
- 9:52Why is this bad
- 10:28The truth
- 10:49Human errors
- 11:32The Internet
- 12:57BGP Config
- 14:10Single Data Center
- 16:03Dog Ruby
- 16:59Database
- 20:28Application Server Capacity
- 21:16Metrics Always Be Measuring
- 22:09Instrumented Applications
- 22:27Campfire Bot
- 23:23Tally
- 24:43HAProxy
- 25:09Troubleshooting
- 25:35MySQL Roll Swap
- 26:18Intermission
- 26:46Maintenance
- 27:05Schema Change
- 27:24Human Factors
- 28:07Blameless Postmortem
- 28:50Recap
related talks
56mGetting all the 99.99(9) you always wanted
40mStrategies for Zero Down Time, Frequent Deployments
29mParticular Particulars about "Zero Downtime Deploys"
56mHow to lay the groundwork in advance, or the path to 99.99% project uptime
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
46mMeeting Impossible SLAs: How we made our uptime 99.999%
47mUptime == Money: High Availability
30mHow are we transforming the org to achieve 99.99% uptime at DigiCert