Mastering stream-to-stream JOINs

on YouTube ↗32m

" summary

Hazelcast presents Jet, a SQL stream-to-stream join engine that implements time-bounded joins by using time-ordered heaps instead of buffers and adding a timestamp-with-key to unify per-stream clocks. The engine repartitions streams by join key or broadcasts right-side streams, ignores late events, garbage-collects buffers when cluster-coordinated watermarks (backed by Raft state) advance, and has been tested at tens of thousands of events per second with the algorithm rewritten and deployed end-to-end.

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 1:52Hazelcast overview
3. 4:02Jet stream processing
4. 7:17Practical JOIN example
5. 10:30Distributed JOIN challenges
6. 13:46Timestamp sync
7. 16:14Optimizing JOIN with timestamps
8. 19:05JOIN types and implementation
9. 20:40Q&A
10. 22:47Time types and buffer config
11. 25:06Data storage and queries
12. 26:34Query execution
13. 27:17Data throughput
14. 28:02Hazelcast scaling issues
15. 29:27Hazelcast tips
16. 30:07Hazelcast architecture and future

" related talks

• 55mHazelcast - Hitchhiker’s Guide. Grzegorz Piwowarek. Java Fest 2020 OnlineGrzegorz Piwowarek · Java Fest 2020
• 01h 13mHow We Built SQL in HazelcastVladimir Ozerov · Joker 2020
• 45mEasy Scaling with Open Source Data StructuresTalip Ozturk · GOTO Chicago 2015
• 29mEnhancing Vector Databases with Real-time Stream ProcessingRaj Barua · Great International Developer Summit 2024
• 41mParallel and distributed big joins in H2OMatt Dowle · Data By the Bay 2016
• 05mStartup Showcase: HazelcastMiko Matsumura · Strata Conference New York + Hadoop World 2013
• 21mStream Processing and SQLBruno Bonnin · ScalaIO 2018
• 01h 12mHidden pearls for high-performance-persistence in JavaSven Ruppert · Joker 2020