Keynote | Flink Ahead: What Comes After Unified Batch & Streaming?
summary
Flink claims unified batch/streaming now handles petabytes per day with SQL, and community releases cut TPC-DS (10 TB on 20 machines) runtimes by about 3x overall, with Flink 1.11 attracting ~270 contributors for that release. The runtime adds pipeline region-based scheduling (Flink 1.12) to mix pipelined and blocking exchanges, and work unifies DataStream operators so bounded runs avoid RocksDB’s ~10x slowdowns by sorting plus a single-key state backend. A savepoint handoff pattern and Kubernetes HA without ZooKeeper are also delivered.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:10Mandatory Recap
- 1:27Flink in the Software Foundation
- 2:59Github
- 4:33Highlights
- 5:47Comparison Baseline
- 7:03Flink 112 Benchmark
- 8:22NextMark Benchmark
- 9:29Thank You
- 10:14Main Topic
- 11:42Recap
- 12:55Unified API
- 15:31Flink query plans
- 16:54Pipeline regionbased scheduling
- 21:05What is the next step
- 22:30Running a data stream program on unbounded input
- 23:28Why does the data stream API have the problem
- 24:22What happens if we do this naively
- 27:09Unified operators
- 27:44Mixed data
- 28:59Batch vs Streaming
- 31:26Unified Batch Streaming
related talks
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
38mApache Flink: A Very Quick Guide
42mStream Processing beyond Streaming Data
39mA deep dive into Flink SQL
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
41mBeyond the Hype: When Apache Flink Solves Real Problems
47mBring Flink to Large Scale Production
43mLeverage the power and simplicity of SQL on Apache Flink