The art of the live upgrade - 10 yrs of evolving a live system
summary
Klarna’s main payment system runs 24/7 for over a decade with no scheduled downtime, evolving via frequent live upgrades that push 2–12 tickets per day, handle 250 apps and about 4,000 modules, and keep ~99.982% yearly availability. The pipeline uses git/Bitbucket, Jira, Jenkins, Docker, Ansible, AWS, RPMs and Artifactory, nightly 4–5 hour system tests, dynamic MD5-based module detection, atomic load and purge actions, shadowing and gradual traffic shifts, and code migration via qualified tail calls — a working, battle-tested workflow.
AI-written summary · about
chapters (33)
- 0:00Intro
- 2:37Kred system vital stats
- 3:53Pipeline and Branching model
- 5:32Problems in the pipeline
- 6:35Quick fixes and configuration changes
- 7:44Keeping developers happy/productive
- 12:06Splitting up bad supervision trees
- 13:04Reducing coupling
- 16:39Enable small changes without stopping
- 17:44We don't use OTP releases...
- 19:31Fundamental Design Decisions of Erlang
- 20:26Loading an Erlang module
- 20:55Loading a new version (atomic)
- 21:13New callers will use the new version
- 21:59Code pointers
- 23:14The Code Server: code:(...)
- 23:57Basic code operations
- 25:41Fully purging a module
- 25:53Detecting modified modules
- 26:46The Code Primitives: erlang: (...)
- 28:39Atomic load of multiple modules
- 30:27Beam code and Erlang VM compatibility
- 32:26Traditional software code scenarios
- 33:00Live upgrade code scenarios
- 33:40Beware systematic failure on all nodes
- 35:30Sanity Checks
- 35:48Separate releasing from activating
- 36:24Separate may-crash from must-not-crash
- 37:48Temporary Dirty Hacks
- 38:30Full upgrade testing
- 39:30Remember to enable code migration
- 40:30Remember to update settings everywhere
- 41:07Advantages and disadvantages
related talks
51mContinuous Deployment for a Billion Dollar Order System
33mThe Road to Enable DevOps Within the Payment Industry
25mAutomating Kubernetes Upgrades for Zero Downtime
34mZero Downtime Database Migrations and Deployments
22mFrom Code Commit to Production Within a Day
24mContinuous Deployment at Scale
06mLightning Talk: Keeping the Show Running: Backstage Maintenance from a Platform Eng...
17mThe (IT) transformation of KNAB: transforming a scale up for the future