Scale Your Batch / Big Data / AI Workloads Beyond the K8s...
Talkat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 17:00· Pavilion 7 | Level 7.3 | Room S03·
summary
Runs a 1,000-job experiment (each job with 10 pods, 3–5 minute work, 15 minute deadline) on Quark-simulated nodes and shows the default Kubernetes scheduler produces many partial allocations causing a 51% failure rate (worse when parallelism increases to 81% failure rate). A co-scheduling plugin fixes all-or-nothing allocation but blows up end-to-end latency by about an order of magnitude. Kueue-style queue managers hit near-optimal throughput — about 50 minutes total completion in the test — while MCAD, Volcano and YuniKorn trade off CRD changes, placeholder pods, or custom schedulers for different levels of flexibility and node-aware allocation.
AI-written summary · about
chapters (14)
- 0:00Scaling batch AI workloads
- 1:07Offline vs online workloads
- 4:17Scaling batch jobs on K8s
- 8:08Resource fragmentation in K8s
- 10:13Co-scheduling plugin limits
- 12:00Job scheduling managers like Q
- 14:31Custom schedulers vs job managers
- 16:16Coordinator features
- 18:05MCAD for flexible workloads
- 19:51Volcano custom scheduler
- 21:16Unicorn placeholder pods
- 23:01Q extensible job manager
- 24:28Why Q was chosen
- 25:43Q&A workload prediction
related talks
25mScale your Batch / Big Data / AI Workloads Beyond the Kubernetes Scheduler
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
24mKubernetes Batch Processing at Scale - A Scheduling Perspective
26mIntro: Kubernetes SIG
48mScale and Performance Testing of Kubernetes: Answers for Specific Applications
31mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I]
07mHow to Handle Fair Scheduling in a Private Academic K8s infrastructure
25mKube-scheduler-evaluator: Evaluating Kubernetes Schedulers at Hyperscaler…