Lightning Talk: Mind the Topology: Smarter Scheduling for AI Workloads on Kubernetes
summary
KAI is a Kubernetes-native scheduler that plans whole distributed AI workloads instead of placing pods one-by-one, claiming measurable latency and throughput gains by aligning pod placement with network topology. It models group-level communication (examples: prefill and decode groups), searches for placements across racks and zones to keep heavy-communicating pods together, and reduces cross-rack KV cache transfers. Open-sourced as Kai scheduler, it integrates with existing workload managers and targets GPU- and network-bound inference and training pipelines.
AI-written summary · about
chapters (6)
- 0:00Kai scheduler for AI workloads
- 1:05Topology scheduling challenges
- 3:37Workloads and network topology
- 5:51Disaggregated inference pipeline
- 8:23Kai pod placement optimization
- 9:06Call to action
related talks
38mBeyond Kubernetes: Adapting To Spe...
24mAccelerate Your AI/ML Workloads With Topology-Aware Scheduling in Kueue
30mInstant KAI Sandboxes with vCluster
30mTopology-Aware Scheduling for AI Training & Inference With Kueue
18mKeynote: Optimizing AI Workload Scheduling: Bilibili's Journey To an Efficient Cloud Native AI Platform
09mLightning Talk: Sailing Kubernetes Operation with AI Power
09mLightning Talk: Optimize the Ray Schedule and Autoscaling on Kubernetes Cloud: A Hetero...
06m⚡ Lightning Talk: Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization