Building Pipelines at Scale
summary
A reproducible approach turns Jenkins jobs into code: Puppet-managed job templates generate pipelines from a pipeline.pp manifest, producing per-customer jobs and promotions and allowing full Jenkins rebuilds in minutes. Problems: editing XML is brittle and jenkins reloads cause brief unavailability. Alternative lighter approach uses build flow + job generator to create dynamic, parallel pipelines on the fly and job config history for diffs. Outcome: scalable, reproducible pipelines for dozens of projects, faster deploys but still requires monitoring, retries and plugin-aware maintenance.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:16Building Pipelines at Scale
- 4:19Example: a puppet pipeline
- 6:32Ok, let's use pipelines!
- 7:26How to scale?
- 9:10Infrastructure as Code
- 10:57Modules structure
- 11:48Structure of the module
- 20:45Advantages of setting pipelines in Puppet
- 25:32Disadvantages
- 27:07A lighter approach: Jenkins plugins
- 28:35Plugin #1: Build flow
- 30:07Build Flow: Groovy scripts
- 35:58Plugin #2: Job Generator
- 37:17Other bottlenecks/problems at scale
related talks
04mContinuous Delivery with Jenkins' New Declarative Pipeline & Artifactory
34mContinuous Delivery with Jenkins Workflow
48mWorkflow in Jenkins
59mJenkins & continuous delivery pipeline
06mBuilding Continuous Delivery Pipelines
33mJenkins 101
45mBecoming a Plumber: Building Deployment Pipelines
29mFrom scheduled builds to commit triggered pipelines