SLA Aware Batch Scheduling in Apache YuniKorn with Multi-Tenant Preemption
Talkat Kubernetes Batch + HPC Day Europe 2023 · on Tuesday, 2023-04-18· at 15:50· Hall 7, Room A | Ground Floor | Europe Complex·
summary
Apache YuniKorn implements SLA-aware batch preemption using hierarchical queues with guaranteed and max capacities (example: org 400MB, leaf max 200MB, guaranteed 100MB), queue preemption fence boundaries to stop cross-tenant theft, and an opt-out annotation allo preemption=true. Preemption waits by default 30s preemption delay, prefers short-lived executors, never preempts same-application tasks, keeps queues above guaranteed capacity, and treats system queue pods as protected; demo uses 100m CPU / 100MB pod requests.
AI-written summary · about
chapters (11)
- 0:00Apache YuniKorn intro
- 1:52YuniKorn core features
- 2:57Batch workload preemption
- 5:06K8s multi-tenant limits
- 7:13YuniKorn preemption design
- 9:04YuniKorn preemption rules
- 12:15App-aware preemption
- 13:41Queue fencing & protection
- 15:05Configuring preemption
- 17:36Live demo: preemption
- 23:55YuniKorn future & thanks
related talks
30mApache YuniKorn A Kubernetes Scheduler Plugin for Batch Workloads
41mApache Yunikorn: State of Union and Future
36mNext Level Spark on Kubernetes with Apache YuniKorn (Incubating)
38mEnergize Multi-Tenancy Flink on K8s with YuniKorn
28mMultitenancy and Fairness at Scale with Kueue: A Case Study
33mBatch Systems in Production with Kueue: Multi-Tenancy and Fungibility
32mA Comparative Analysis of Kueue, Volcano, and YuniKorn
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers