Topology-Aware Scheduling for AI Training & Inference With Kueue
summary
Kueue proves capable as a quota-and-scheduling layer for a migration from bespoke infra to Kubernetes but requires concrete TAS extensions: a multi-layer TAS that composes topology CRs, does a bottom-up capacity rollup and a top-down best-fit slice placement, and integrates with external admission (TPU slice controller). It handles TPU cubes represented as 64-chip / 16-node cubes and stitched shapes >2,000 TPUs, supports NVL72 racks (up to 72 GPUs, 18 nodes) and slice multiples (16, 32), and shows workable but fragmentation- and taint-related scaling tradeoffs.
AI-written summary · about
chapters (9)
- 0:00Introduction to Kueue
- 1:34Kueue core capabilities
- 2:42Topology-aware scheduling
- 5:07GKE TPU v7 integration
- 8:02Meta case study
- 10:52Multi-layer topology constraints
- 16:54How multi-layer scheduling works
- 20:06Scalability and limitations
- 23:53Summary and future directions
related talks
24mAccelerate Your AI/ML Workloads With Topology-Aware Scheduling in Kueue
28mMultitenancy and Fairness at Scale with Kueue: A Case Study
10mLightning Talk: Mind the Topology: Smarter Scheduling for AI Workloads on Kubernetes
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
38mBeyond Kubernetes: Adapting To Spe...
28mScaling LLM Workloads on TPUs with Ray: From Pre-Training to Inference
34mDeep Customized Kubernetes for Machine Learning in Tencent
40mAI Training & Inference on Kubernetes: Compute Patterns and Best Practices with Karpenter.