Towards Flink 2.0: Unified Batch & Stream Processing
summary
Flink 1.9 delivers concrete steps toward a unified batch/stream stack: the Table API runs on the DataStream runtime via the Blink planner, sources move to a single model with FLIP-27 unified source (bounded splits that can be open-ended), operators gain selectable push-based operators so batch-style pulls are possible, recovery uses region cuts plus a pluggable shuffle to avoid full restarts, programmatic job control and executors (FLIP-74, FLIP-3073) improve submission and lifecycle, and stop-with-savepoint fixes two‑phase commit races.
AI-written summary · about
chapters (17)
- 0:00Intro
- 2:02Batch Processing is a special case of Stream Processing
- 2:25The remaining 40% of the truth
- 3:35Real-time Stream Processing
- 4:02Stream Re-Processing
- 4:19Batch-style Processing
- 4:52Batch and Streaming Processing Styles
- 8:34A Typical Unified Use Case: Bootstrapping State
- 9:56Future of the DataStream API
- 11:04Exploiting the Batch Special Case
- 14:37A New unifieds Source Interface
- 16:44Table API/SOL
- 19:02Push-based and Pull-based Operators
- 22:15Scheduling Strategies
- 26:13Configuration All the Way Down
- 27:16A new Executor Abstraction for Executing Jobs
- 33:01What else is the community working on?
related talks
45mThe State of Flink and how to adopt Stream Processing
38mFrom Flink 1.3 to 1.4
30mBlink: Improvements to Flink & its application in Alibaba Search
35mBeyond the Watermark: On-Demand Backfilling in Flink
41mConcurrent Stream Processing
47mAll the DataOps, all the paradigms
36mMicroStream Labs – New Incredible Feature Preview
17mDatabricks LakeFlow: A Unified, Intelligent Solution for Data Engineering