What Kind of CPU is it Anyways? Airbnb's Journey to Heterogeneous Clusters
Talkat KubeCon + CloudNativeCon North America 2021 · on Wednesday, 2021-10-13· at 14:30· Room 408 AB + Online·
summary
Airbnb migrates Kubernetes from many single-instance-type clusters to mixed-instance clusters using EC2 ASGs and cluster autoscaler, cutting operational cluster types from ~90 toward a smaller fleet and enabling faster instance upgrades. Key fixes: remove instance-type node selectors and introduce generic pause pods (pause pods handle spare capacity and are evicted ~98% of the time), add missing topology zone keys to ASG templates, and prototype a custom expander plugin (gRPC service) to pick instance types by price, availability, and bin‑packing. Performance tests show latency within ~10–20% across instance types; rollout is in progress across ~50–60 clusters.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 0:32Outline
- 1:36Brief History of Kubernetes
- 8:13Nobody knows whats going on
- 9:12The unhappy path
- 12:40Single Instance Type
- 14:40Single Instance Type Modification
- 15:10Pause Pod Modification
- 17:40Custom Expander Plugin
- 21:19Organizational Issues
- 21:39Migration Fatigue
- 22:11Customer Concerns
- 23:39Performance Testing
- 24:53Very Latency Testing
- 25:22Technical Issues
- 25:49Pod Topology Spread
- 26:14Logs
- 27:15Empty ASGS
- 27:46Cluster Autoscaler
- 28:40How does Scheduling Simulation work
- 30:04Topology Zone Key
- 30:19Short Term Fix
- 31:45Summary
- 32:57Future Proof
- 33:10Cost
- 33:47Next Steps
- 34:23Existing PRs
- 34:40Conclusion
- 34:50Thank You
related talks
26mThe Road To Cilium: Migrating 150+ Kubernetes Clusters at Airbnb
27mScaling K8s Clusters Without Losing Your Mind or Money
30mScaling Kubernetes to Thousands of Nodes Across Multiple Clusters, Calmly
33mUncover Kubernetes cost-optimization trends
21mWhy should you bother about cluster autoscaling- Karpenter
31mDeclarative Underlays: Scaling Purpose-Built Infrastructure(Clusters)...
31mAccelerate AI adoption: automated cloud efficiency
29mDid Kubernetes Make My p95s Worse?