The Chicken or the Egg: Creating a Stable Preprod...
summary
Built a full, isolated production replica called TestCloud using a two-phase double bootstrap: two bare metal management hosts bootstrapped with Ansible and libvirt, then core control plane deployed via Terraform and Salt, Vault and a layered pillar model. A six-person team runs LDAT automation to manage circular dependencies (Ansible for networking, Salt for config), cuts new DC deploys from ~9 to 3–4 months, runs 7–8 concurrent deploys, serves 500+ customers, achieves ~90–95% parity and catches many bugs before production.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 1:02Previous test data center problems
- 2:27Need for new pre-prod
- 3:30Scaling rapid deployment
- 4:52LDAT for automation
- 6:41Test cloud team constraints
- 8:05New test cloud requirements
- 9:29Key terms and tools
- 12:18Double bootstrap approach
- 13:01Bootstrapping hardware
- 15:26Control plane system classes
- 17:13Provisioning first compute host
- 18:20Setting up bootstrap host
- 19:25Terraform for core infra
- 21:29Terraform lessons learned
- 23:18Naming and scaling infra
- 24:00Salt for config management
- 26:27Deploying compute hosts
- 27:50Non-core infrastructure
- 29:14Stable production-like env
- 31:20Purpose of stable env
- 32:22Experiences after TestCloud
- 33:47Onboarding teams
- 35:12Maintaining stability
- 36:16Key wins and lessons
- 37:41Cost of repo forking
- 38:23Takeaways and advice
- 39:06Q&A
related talks
33mDeveloping for Multi-Component Environments While Keeping Your Sanity
27mFrom zero to prod with Nomad
27mAdvanced Multipurpose Infrastructure: A Self-Managed Cloud Environment with HashiCorp Terraform
49mAutomation in Deployment on Hybrid Hosting and Private Cloud Environments
22mHow OpenDev Tests Complete System Deployments with Zuul and Ansible
42mArchitecting for Reliability
32mAutomating and Testing Production Ready Kubernetes Clusters in the Public Cloud
25mContinuous Delivery and Testing in Production