Creating Multistage Development Environments with a K8s Cluster of 8000 Pods and Growing
summary
A multistage development platform for about 100 microservices running on an AWS Kubernetes cluster sized for 8,000+ pods, built using kops and Terraform. The implementation reduces per-environment configs from ~1700 to ~70–80 via templates, uses daemonset proxies and switches networking from iptables to ipvs to avoid routing slowdowns, combines spot and on‑demand node groups to handle spot evictions, automates per‑service migrations, and provides fast ephemeral environments that spin up in 2–3 minutes while shrinking server count from 1,600 to 60.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:08Why Kubernetes
- 3:16Traditional infra problems
- 4:43Choosing Kubernetes & EKS
- 6:06Cluster setup with KOPS
- 8:18KOPS issues & fixes
- 10:28Service migration prep
- 11:57Simpler configurations
- 13:03Developer motivation
- 13:44Migration start & early wins
- 14:06Black Friday load issues
- 15:29Monitoring problems & fixes
- 16:11IPVS critical issue
- 19:19Amazon API limits
- 20:22IPVS-less replacement results
- 21:04Migration outcomes & scale
- 22:08Automating deployments
- 23:16Q&A: Dynamic environments
- 26:07Q&A: Infrastructure automation
- 27:33Q&A: Scaling & modules
- 28:57Q&A: Backup & recovery
- 30:30Q&A: Change testing
- 31:53Q&A: Data & databases
- 33:00Summary
related talks
35mKubernetes journey
36mOptimizing Resource Usage in Kubernetes
26mMulti-Tenant vs Micro-Clusters in Kubernetes-Driven Platform Engineering
28mBuilding K8s on AWS & Enabling Developers to Deliver. What to Do Better - Greg Sansom & Fahad Tariq
35mTo Infinite Scale and Beyond: Operating Kubernetes Past the Steady State
22m10 Kubernetes is hard!
34mScaling Cloud Development Environments on K8s: Enterprise-Scale Adoption Lessons
29mRunning Kubernetes on 'small scale' clusters - lessons learned