Flink’s New Batch Architecture
summary
Flink adds a region-based failover that restarts only affected pipeline regions (feature in Flink 1.9), plus a pluggable scheduler and a dynamic-slot model (FLIP-56) where task executors advertise resources and the resource manager slices slots to avoid wasted memory. Alibaba prototypes a dynamic group scheduling plugin using control-edge hints (concurrent, sequential, start-on-finish) and an external blocking shuffle with hash and sort-merge modes, disk-aware placement and a priority read queue; tested at ~1,000-node scale and reduces long-tail IO and idle task managers.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:28Everything Is a Stream
- 2:37Batch Processing
- 3:49Batch vs. Stream Processing
- 4:36Executing a Flink Program
- 6:06Current Batch Scheduling
- 8:02Advanced Batch Scheduler
- 9:02Making the Scheduler Pluggable
- 10:22Resource Management for Batch Jobs
- 12:00Fine Grained Resource Management
- 13:39Flink's Batch Failover
- 16:18Fine Grained Recovery in Action
- 17:44How to Use Flink with Fine Grained Recovery?
- 18:57The Importance of Shuffling Data
- 20:17Pluggable Shuffle Service
- 22:02Dynamic Group Scheduling Strategy
- 28:12External Blocking Shuffle Service
- 35:25Conclusion
related talks
44mNews from Flink’s engine room: “Full steam ahead”
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
31mRethinking of fault tolerance in Flink: what lies ahead?
42mApache Flink deep-dive #bbuzz
20mAutoscaling Apache Flink Applications
43mDynamic Scaling - How Apache Flink adapts to changing workloads
33mFlink and Beam: Current State & Roadmap
36mApache Flink in the Cloud-native Era