Running Kubernetes on Ephemeral Infrastructure
summary
Runs Kubernetes clusters that destroy and rebuild every 48 hours, touching ~10,000 EC2 instances daily across 2,500–3,000 nodes and serving >1M pods per day. Implements a controller that filters safe-to-terminate nodes and a pre-termination actions framework that calls leader step-downs, moves VPC routes, and injects iptables resets. Switches from Calico to AWS VPC CNI for pod IPs, adds local image caches, changes Vault to sign host CSRs, and reports big win on faster patching and reduced attack dwell time with operational tradeoffs.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:40About Palantir
- 2:50About Ephemeral Infrastructure
- 3:33Scaling Limitations
- 4:57Ephemeral Infrastructure
- 6:01Kubernetes in Production
- 7:52Benefits
- 10:37Challenges
- 11:08Leaders
- 11:33Pre termination actions framework
- 12:16Stepdown endpoint
- 13:12Cloud challenges
- 14:26Bad hardware
- 15:39Host boot up
- 18:02Ingress and Egress
- 19:49Networking
- 23:58Key takeaways
- 25:17Vendor showroom
- 25:58Longrunning jobs
- 27:47Storage
- 29:19Pipelines
- 30:23Problems
- 32:55Monitoring and alerting
- 35:13Managing images
- 37:24DNS issues
related talks
06mKubernetes clusters should be ephemeral
30mMission Critical Kubernetes
47mAnother Cluster Bites the Dust... and That’s Just Fine!
36mOptimizing Resource Usage in Kubernetes
34mCreating Multistage Development Environments with a K8s Cluster of 8000 Pods and Growing
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
44mEssential-8 Security in Practice with Kubernetes
35mKubernetes journey