Taking a look under the hood of Apache Flink’s relational APIs
summary
Flink exposes a Table API and SQL that both translate into the same Calcite logical plan, run through a Volcano-style cost-based optimizer and then into DataSet/DataStream programs with Apache Calcite handling parsing/validation and code generation producing Java functions compiled on the cluster. Batch queries already support joins, sorts and many SQL functions; streaming is getting streaming window aggregates (prototype working) and joins later. Cost estimates don’t use source statistics yet, so join order is currently preserved.
AI-written summary · about
chapters (17)
- 0:00Introduction to Flink relational APIs
- 3:15Motivation for relational APIs
- 8:13Batch vs streaming relational queries
- 11:29Table API overview and example
- 14:20SQL API overview and example
- 17:36Translation architecture overview
- 20:24Logical plan and validation
- 24:44Optimization process
- 27:55Code generation and execution
- 32:15Current state and limitations
- 33:22Future plans for streaming support
- 36:57Summary and benefits
- 38:04Q&A: Why Table API?
- 40:30Q&A: Optimizers and cost functions
- 42:13Q&A: Table scope and state
- 43:57Q&A: Flink vs Kafka Streams
- 45:43Q&A: Multi-window queries
related talks
39mA deep dive into Flink SQL
43mLeverage the power and simplicity of SQL on Apache Flink
38mApache Flink: A Very Quick Guide
42mWhat's new for Flink's Table & SQL APIs? Planners, Python, DDL
37mSQL Ask-Me-Anything
39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
41mBeyond the Hype: When Apache Flink Solves Real Problems
46mFlink SQL in 2020: Time to show off!