A beginner's journey: Scaling self-hosted platform infrastructure with Nomad
summary
Built a self‑hosted image factory that runs Packer-driven builds for AWS, Azure and vSphere and executes ephemeral containerized GitHub Actions runners on Nomad clusters deployed to three Intel NUCs plus laptop and Raspberry Pi, with images pushed to a private registry and HCP Packer metadata consumed by Terraform. Scaling uses a Nomad autoscaler tied to Datadog metrics and a GitHub listener that creates per‑run jobs, cutting a full workflow from 34 to 20 minutes; an early run produced 113 unintended builds during testing but the system otherwise works.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:46Journey
- 4:50My end goal
- 6:16Why I chose Nomad
- 8:42Personal reasons
- 10:44Solution overview
- 12:37Containerizing runners
- 17:16Building an image factory
- 18:59HTTP Packer
- 22:27Scaling options
- 27:48Future plans
related talks
23mNomad the Easy Way - Creating GitHub & GitLab Pipelines in Minutes
27mFrom zero to prod with Nomad
33mAbstracting Nomad: Patterns for Deploying to Nomad at Scale
24mHow we Orchestrate 7M+ Builds/Month with Kubernetes - Danielle Tomlinson
39mBuilding Efficient Continuous Integration Pipelines with Nomad, Vault and Jenkins
01h 06mSchedule AI workloads efficiently on Nomad with node pools
30mI just want to ship my code. Waypoint, Nomad, and other things.
35mLiving the Nomadic life