How to Replace a Jet Engine of your System In Flight
summary
A distributed build system migrates from direct client->scheduler->worker calls to a new design with a persistent queue, separate build-artifact and build-progress services, and the same distributed client library; rollout uses Borg-managed, incremental rollout across zones with side-by-side metrics. Migration requires writing throwaway bridge code to handle many intermediate states, supporting old client versions, validating new backends can handle ~45,000 commits/day and ~5 million build/test requests producing petabytes, and using checklists and rehearsals — result: zero downtime deployment.
AI-written summary · about
chapters (7)
- 0:00Introduction to build systems
- 3:43The distributed build system
- 6:06Scaling and architectural needs
- 9:28Migration strategy
- 16:33Decoupling and throwaway code
- 20:13Incremental rollout and metrics
- 23:49Practice and checklists
related talks
32mChanging Engines Inflight - A Real Life Modernisation Story
41mSurviving Continuous Deployment in Distributed Systems
53mThe Road to Continuous Deployment: a Case Study
24mArchitecting the Unconventional: A Case Study in Software Design
35mRedundancy Driven Development - Navigating disruptive codebase changes.
30mBuilding a Distributed Build System at Google Scale
35mThe importance of Engineering Systems: How to Build Efficient ...
52mReplacing the Gearbox (while Driving Down the Motorway)