Boosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
Talkat Cloud Native + Kubernetes AI Day NA 2024 · on Tuesday, 2024-11-12· at 14:05· Salt Palace | Level 1 | Grand Ballroom A·
summary
ByteDance claims catalist raises LLM training and inference throughput by adding topology-aware scheduling on Kubernetes: it implements die-level affinity and memory-bandwidth anti-affinity for high‑density CPUs (AMD Genoa up to 192 cores), creates job-level inter‑RDMA affinity at the ToR level to avoid switch congestion, and enforces GPU‑RDMA affinity at the PCIe‑switch level to enable GPUDirect RDMA; scheduler changes include a pod‑group prefilter/postfilter and an out‑of‑band resource manager replacing kube CPU/memory managers for accurate NUMA/CCX allocation.
AI-written summary · about
chapters (9)
- 0:00Topology-aware scheduling intro
- 1:50LLMs and Kubernetes background
- 4:02Catalyst resource system
- 6:38Kubernetes Topology Manager limits
- 10:57NUMA-aware AI scheduling
- 13:52NUMA pinning policies
- 15:43TOR-level RDMA affinity
- 17:52PCI switch GPU-RDMA affinity
- 18:38Catalyst roadmap
related talks
24mImproving GPU Utilization and Accelerating Model Training with Scheduling Framework and NRI
34mUnlocking Performance: Topology-Aware CPU Scheduling W...
24mAccelerate Your AI/ML Workloads With Topology-Aware Scheduling in Kueue
40mBuilding a Fine-Grained and Intelligent Resource Management System on Kubernetes
19mML training acceleration with heterogeneous resources in Bytedance
30mTopology-Aware Scheduling for AI Training & Inference With Kueue
38mBeyond Kubernetes: Adapting To Spe...
18mLet There Be Topology-Awareness in Kube-Scheduler!