Life in the Trenches of healthcare.gov
summary
Deployment stories with timestamps and root causes not abstracts: permission changes at 10:48 a.m. revoke the app user and cause 100% errors while latency collapses; identity proofing uses Experian and state is kept in an XML database behind Oracle; storage relies on 19 NetApp filers that remount read‑only after a 30s write denial forcing manual reboots across 171 hosts and choreographed restarts; the accounts/auth subsystem ships with a 25% error rate until a manual load‑balancer switch fixes it.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:47Healthcaregov Failure
- 4:48Monitoring
- 6:26Architecture
- 8:16Database
- 8:44High error rate
- 9:48Low latency
- 10:46Permissions changed
- 11:50Three possibilities
- 13:49Another incident
- 15:35Why the site started serving errors
- 16:39A single point of failure
- 18:05Monitoring is low
- 18:35Ready for traffic
- 19:19What does it mean
- 19:40A story
- 22:43Taking the blame
- 23:20Production as afterthought
- 24:49Conclusion
