Energize Multi-Tenancy Flink on K8s with YuniKorn
summary
Runs Flink on Kubernetes with YuniKorn as a replacement (or secondary) scheduler, using Helm and an admission controller or labels to identify Flink applications in standalone, operator or native (Kubernetes RM) modes. Claims: doubles scheduling throughput in tests that place 50,000 pods across 2,000–4,000 nodes, enforces resource fairness across namespaces with hierarchical queues and FIFO/fair policies, supports preemption, bin‑packing vs spread placement, autoscaler integration and a UI, and is production usable.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:26Flink Kubernetes
- 3:14Why Kubernetes
- 5:06Is Flink perfect
- 5:35Challenges
- 7:13How it works
- 9:13Flink on K8s
- 13:37Unicorn
- 14:24What is Unicorn
- 15:15How Unicorn works
- 18:00Unicorn features
- 19:52Unicorn online convenience
- 27:42Studying policy
- 29:27Resource fairness
- 31:08Evaluation
- 31:58Schedule
- 33:18Unicom
- 34:49UI
- 35:44Roadmap
related talks
30mApache YuniKorn A Kubernetes Scheduler Plugin for Batch Workloads
36mNext Level Spark on Kubernetes with Apache YuniKorn (Incubating)
41mApache Yunikorn: State of Union and Future
30mBeyond Experimental: Spark on Kubernetes
37mSpark and Flink Running Scalable in Kubernetes #bbuzz
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
42mManaging Apache Flink on Kubernetes - FlinkK8sOperator