Connecting Apache Flink and the Pravega Stream Store

on YouTube ↗46m

" summary

Pravega is a stream storage that stores unbounded streams by splitting them into mutable segments that can auto-scaling segments through split and merge and persist data using Apache BookKeeper for low latency and a configurable tier two (HDFS NFS extended S3). Connectors implement a source that uses the Pravega reader group and controller to produce aligned checkpoints with Flink and a sink that uses Pravega transactions to guarantee exactly-once semantics. Integration is implemented and demonstrated, with scaling signals and HA uses under development.

AI-written summary · about

" chapters (17)

1. 0:00Introduction
2. 1:38What is Stream
3. 4:05Processing Streams
4. 7:40Pravega
5. 15:45Segment Manipulation
6. 16:35API Semantics
7. 17:40EventDriven API
8. 18:54Operation semantics
9. 21:14Writers crashing
10. 24:27State synchronizer
11. 27:53Checkpoints
12. 29:43Takeaways
13. 31:53Stephan Ewen
14. 37:53Writing to Pravega
15. 39:13How the transactions work
16. 41:21Example
17. 44:37Next steps

" related talks

• 51mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Voxxed Days Ticino 2025
• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 32mKeynote: Apache Flink - Completing Cloudera’s End to End Streaming PlatformMarton Balassi, Joseph Witt · Flink Forward Virtual 2020
• 30mFlink + Pulsar: The Path To Unified Batch and StreamingAddison Higham · Flink Forward Global Virtual 2020
• 42mStreamhouse Architecture with Flink and PaimonAlexey Novakov · Big Data & RPA Conference Europe 2024
• 40mThe Seven Deadly Sins of StreamingGiannis Polyzos · Big Data Conference Europe 2025
• 25mTips and Tricks for Developing Streaming and Table ConnectorsEron Wright · Flink Forward Europe 2019
• 01h 01mApache Fluss: A Streaming Storage for Real-Time LakehouseJark Wu · Future Data Systems Seminar Series (Fall 2025)