Everything is connected: How watermarking, scaling, and exactly once impact one another in Pravega

on YouTube ↗42m

" summary

Pravega implements a Flink source/sink connector that ties stream cuts and a reader group to Flink checkpoints so watermarks only advance when transactions commit; watermark objects are triples (lower time, upper time, stream cut). Scaling (dynamic segment splits/merges) changes parallelism and forces checkpoint, transaction and watermark coordination. The connector uses Flink checkpoints to trigger Pravega checkpoints and commits transactions for exactly-once via transactions. Demo ingests 50 files (0–49); watermark reaches 49, showing it works.

AI-written summary · about

" chapters (30)

1. 0:00Intro
2. 0:55Pravega: Storage for data streams
3. 2:43Landscape
4. 6:02Drone video streams + telemetry
5. 8:15Data streams: Paralelism
6. 8:44Data streams: Workload variations
7. 9:33Data streams: Unbounded
8. 10:26Data streams: Read scalability
9. 10:58Data stream: Cloud native, storage primitive
10. 11:53The recipe for effective stream processing
11. 14:33Low watermark
12. 16:10Global order
13. 17:19Tracking source position
14. 18:19Time windows: Accuracy
15. 18:38Upon crashes
16. 20:34Checkpointing
17. 21:04Checkpoint complicates sinks
18. 22:09Scaling and rebalancing
19. 25:01StreamCuts
20. 25:36Watermarking (writer side)
21. 27:21Watermarking (reader side)
22. 27:56Connectors
23. 28:13Reader Groups
24. 28:57Pravega time windows
25. 30:04Watermarks
26. 30:27With a Pravega reader source
27. 30:54Upon a checkpoint
28. 31:31Upon a Flink checkpoint
29. 32:31Exactly once with Transactions
30. 34:27Demo setup

" related talks

• 15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful JobsShahid Chohan · Flink Forward Global Virtual 2020
• 26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ FlinkPiotr Nowojski · Flink Forward Berlin 2017
• 38mStreaming Fast and SlowCaito Scherr · Data Con LA 2020
• 34mA journey of scaling high-volume low-latency stateful processingArti Pande · Flink Forward Global Virtual 2020
• 42mApache Beam: A Unified Model for...Kenneth Knowles · Flink Forward Berlin 2016
• 28mA look at Flink’s internal data structures & algorithms for efficient checkpointingStefan Richter · Flink Forward Berlin 2017
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020
• 43mStream Processing and Applications in the Modern AgeStephan Ewen · Flink Forward Europe 2019