Do the holes in Swiss cheese leak abstractions?
summary
A cron-run Django management command opens an atomic transaction that forces routing to the primary, reads users, then makes slow external API calls; an ungraceful deployment kills the process leaving a half-open TCP connection and held read locks, a subsequent migration needs an access exclusive lock and queues behind those reads, web requests time out and the site outages. Remediations include removing the dry-run that routed to primary, adding a row-limit flag and tightening RDS TCP keepalive to two minutes.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:04Octopus Energy and Kraken
- 1:45Kraken support site
- 4:53Leaky abstraction examples
- 7:23Swiss cheese model & software development
- 8:44Kraken architecture overview
- 10:23Kraken outage case study
- 15:07Swiss cheese view of outage
- 15:52Leaky abstractions and Swiss cheese
- 18:15Dry run management command
- 19:21Cheese slices of the outage
- 20:40Process termination
- 22:55TCP network connection
- 24:54Database migrations
- 27:09Post-outage remediation
- 29:16Filling in the holes
- 32:34Conclusions
related talks
32mThe Very Hungry Transaction
21mHigh-Availability Django
34mDistributed Systems - What Can Go Wrong and How to Get It Right
34mZero Downtime Database Migrations and Deployments
51mMigrations - The Hardest Actual Problem in Computer Science
39m30 minutes of Failure
04mPREVIEW: Running Kafka for Maximum Pain
29mMigrating Data Quickly and Safely