Towards Flink 2.0: Rethinking the stack and APIs (...)

on YouTube ↗42m

" summary

Stephan Ewen outlines Flink’s plan to unify its batch and stream processing APIs and runtime by reworking the stack and abstractions. The talk covers the roles and interplay of DataSet, DataStream, and the Table API, a new Flink stack with shared abstractions, unified batch/streaming source implementations, and how runtime optimizations for batch versus streaming differ and may interoperate in future Flink 2.0 designs.

AI-written summary · about

" chapters (9)

1. 0:00Introduction to Flink
2. 6:10Stream processing philosophy
3. 12:02Batch vs streaming trade-offs
4. 17:14Streaming components and operators
5. 23:18Unbounded streams and checkpoints
6. 28:56Operator setups and contributions
7. 33:04Table updates and streaming
8. 37:28Unified collaboration in streaming
9. 40:39Community and future outlook

" related talks

• 42mStream Processing beyond Streaming DataStephan Ewen · ITNEXT Summit 2019
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020
• 49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing ApplicationsStephan Ewen · Scale By the Bay 2017
• 35mFlink's Table & DataStream API: A Perfect SymbiosisTimo Walther · Flink Forward Global 2021
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 31mBlueprints for common stream processing use cases with FlinkKostas Kloudas · Berlin Buzzwords 2018
• 32mBig Data Processing : Where do we go next ?Roshan Naik · Flink Forward Global 2021
• 37mThe Future of Apache FlinkAljoscha Krettek · Flink Forward Berlin 2016