Production Population Control: My Cattle are Rabbits!
summary
Cloud environments spawn many more environments than expected—often 6–9 test stacks per app and sometimes up to 10 production regional/DR variants—so define automatable SLOs as code that use occurrences-based SLOs (count successful requests or jobs) instead of time-slices, add “lights on” PromQL checks for Kubernetes API, pod and node health, and run synthetics (Cooper Healthy, Mabl, Runscope, Ghost Inspector) including ALB unhealthy counts and pod start latency. Assign ownership (per-developer for personal envs, rotating mechanic for shared) and tune targets (example: 60% dev, strict prod).
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:22Evolution of infrastructure environments
- 1:25How many environments?
- 2:35How many prod environments do we nee
- 3:42How many test environments do we need
- 4:56Problem statement
- 5:39Properties of SLOs: 1. Low context
- 6:01Properties of SLOs: 2. Easily tunable
- 6:29Must be automatable (Gitops, lac)
- 7:00What to pick for SLls?
- 8:16Occurrences-based SLOS
- 9:29Lights-on Slls for infrastructure and clus
- 10:36Lights-on SLIs for applications
- 11:49Synthetic SLIs for infrastructure and clus
- 12:46Synthetic Slls for applications
- 15:26Define clear ownership of non-productio
- 17:17Rabbit control
related talks
16mLiving without pre-production environments
58mCultivating Production Excellence
36mDesigning Staging Environments
16mStaging Environments: A Blueprint for Pre-production
38mTempering our Production Practices
28mOpenShift for the Enterprise or What Could Possibly Go Wrong
23mSLO's for Openstack - the API for Engineering Teams
22mYou don’t have a production environment