No gain without pain a story of scaling CI with Zuul
summary
Company CI moves from one brittle scheduler to 9 schedulers and shifts queues and pipeline state into Zookeeper for queues and state, enabling rolling upgrades and eliminating long 20–30 minute restart outages. Work includes synchronous reporting, memory-leak fixes, distributed tenant and pipeline locks, and pipeline-level parallelism. Throughput rises from about 3,000 jobs/hour to roughly 6,000 jobs/hour with steadier run-handler frequency, shorter startup times, and no scheduled maintenance windows.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:48Why CI
- 3:06Why Zuul
- 3:57Big projects
- 4:54Job per hour
- 12:55Technical details
- 14:22Problems
- 15:27Resource contention
- 16:08Multiple schedulers
- 17:56Moving to Zookeeper
- 19:06Pipelines
- 21:08Metrics
- 21:52Results
- 23:16User impact
- 23:50Outro
related talks
19mZuul CI for Kubernetes on the Workday Private Cloud
15mZuul at Volvo Cars
26mFaster, Cheaper, Leaner: Horizontally Scaling a CI Pipeline
25mGetting Started with Zuul
24mUser Report of a Zuul Installation in Production
47mCI/CD Agility and Controlling Pipeline Sprawl
36mRise of the Machines [B]
25mAutomating Kubernetes Upgrades for Zero Downtime