Lessons from Automatic Incident Resolution for a Million Databases
summary
Heroku runs about one million databases across tens of thousands of AWS instances and reduces pages by monitoring the single business metric — whether a customer database is available — then encoding repeatable runbooks as scripts and small, composable automation. Monitoring is verified from multiple regions before action, protected by circuit breaker: >20 instances down within 15 minutes, and exposed via a console for bulk approvals. High‑availability uses standby pairs with streaming replication and automated failover to standby in ~2 minutes. Long‑term trends and per‑database SLOs guide fixes.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:47What is a 12factor app
- 2:57What is 12factor
- 4:40Monitor
- 5:39Runbooks
- 7:52Automation
- 11:26Trends
- 12:36Summary
- 13:39Shirts
- 13:50safeguards
- 14:55page out
- 15:53SLAs
- 16:57When to write automation
- 17:12Concerns with automation
- 19:17Provisioning for failover
- 20:11Circuit breakers
related talks
47mBreakout Session
19mManaging Databases at Scale - AWS
32mManaging Databases in the Cloud in Broken
33mHerding Elephants
51mDatabase Incident Management: When Everything Goes Wrong
14mExpect the unexpected: Building resilience with AWS
51mThe Mother of All Databases
18mManaging Databases in the Cloud is BROKEN!