How to make pod assignment to thousands of nodes every day easier
summary
New Relic runs ~270 clusters with 317,000+ pods and builds a mutating admission webhook called scheduling classes that translates a simple annotation into native node affinity and tolerations so Kubernetes handles placement. It orders classes by priority, is deterministic, skips workloads with custom selectors, and currently limits mutations to affinity/tolerations. Outcome: increased Carpenter adoption and better instance diversity. Tradeoffs: opinionated defaults, narrower fields mutated, and added operational complexity of an admission controller.
AI-written summary · about
chapters (14)
- 0:00Introduction and speaker background
- 1:05Kubernetes scheduling primitives overview
- 2:29Node selectors and affinity explained
- 5:59Taints tolerations and constraints
- 7:03New Relic Kubernetes fleet overview
- 8:27Cluster API and Carpenter provisioners
- 11:21Limitations of native scheduling methods
- 13:48Introducing scheduling classes solution
- 16:16How scheduling classes work technically
- 21:58Benefits of instance diversity in pools
- 24:48Improving developer experience
- 27:17Increasing Carpenter adoption
- 29:25Current roadmap and future plans
- 33:01Q&A session
related talks
30mDeep Dive: Scheduling SIG
34mKubernetes Scheduling Features or How Can I Make the System Do What I Want?
01h 01mKubernetes – going beyond the basics
28mScaling in Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes
59mKubernetes - going beyond the basics
24mK8s Pod Scheduling - Deep Dive
24mKubernetes Batch Processing at Scale - A Scheduling Perspective
52mKubernetes - going beyond the basics