Life in the Trenches of healthcare.gov
summary
Outage stories focus on concrete failures not theory: a permissions change at 10:48 causes permission denied errors that instantaneously drop latency and make the core app return 100% errors; a single NetApp drive failure cascades across 19 NetApp filers, Linux remounts filesystems read‑only after ~30 seconds, and recovery requires manually restarting and remounting services on 171 hosts plus carefully ordered restarts of cache, DB and app; monitoring is fragmented across vendors and capacity planning is absent.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:47Healthcaregov Failure
- 4:48Monitoring
- 6:26Architecture
- 8:16Database
- 8:44High error rate
- 9:48Low latency
- 10:46Permissions changed
- 11:50Three possibilities
- 13:49Another incident
- 15:35Why the site started serving errors
- 16:39A single point of failure
- 18:05Monitoring is low
- 18:35Ready for traffic
- 19:19What does it mean
- 19:40A story
- 22:43Taking the blame
- 23:20Production as afterthought
- 24:49Conclusion
related talks
27mLife in the Trenches of healthcare.gov
26mA Tale of Two Outages
34mDatacenter Fires and Other 'Minor' Disasters
39m30 minutes of Failure
30mIncident Groundhog Day
53mHow our critical systems fail, and what we do about it
27mWhen One Line Took Thousands of Websites Offline
32mSurviving Black Friday - A resilience engineering tale