Towards More Efficient and Adaptive Scheduling for Flink Batch
summary
New Flink batch scheduler implements LazyFromSources plus an Adaptive Parallelism strategy that adjusts task parallelism at runtime by merging small partitions and splitting large ones based on input-size metrics; a microbenchmark drops a MapReduce job from ~8s to <3s by shrinking reduces and downstream parallelism. It also adds Speculative Execution: job master detects long‑tail tasks via processed‑bytes and throughput formula and launches duplicate tasks on other nodes, with production runs showing a slow reduce falling from ~10s to ~1s when a speculative copy finishes first.
AI-written summary · about
chapters (5)
- 0:00Introduction to new scheduler
- 1:55Dynamic parallelism strategy
- 8:11Template parenting in workflow
- 10:20Speculative execution overview
- 14:44Summary and conclusion
related talks
44mNews from Flink’s engine room: “Full steam ahead”
43mDynamic Scaling - How Apache Flink adapts to changing workloads
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
20mAutoscaling Apache Flink Applications
38mFaster and Furiouser: Flink Drift
42mApache Flink deep-dive #bbuzz
39mA deep dive into Flink SQL
15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs