Parallel and distributed big joins in H2O
summary
Radix-sort based joins are parallelized and distributed inside H2O (Java backend, R/Python front ends) so large-key integer joins run in memory across a 10-node cluster (320 cores, ~2 TB RAM) reading from HDFS, using PCG random generators for test data. Implementation avoids hash tables, builds ordered indexes (left index in 116s) and performs binary-merge joins; benchmarks show 1 billion rows in 22s and a full 10 billion rows in ~10 minutes on the demo cluster, and the system returns ordered results correctly.
AI-written summary · about
chapters (10)
- 0:00Introduction and setup
- 2:16Cluster startup and config
- 5:11Loading data from HDFS
- 7:12Parallel data joins
- 11:12Performance monitoring Q&A
- 14:23Generating large datasets
- 19:38Algorithm and use cases
- 25:24Executing 10B row joins
- 28:26Architecture and resiliency
- 36:07Concluding demo and summary
related talks
50mFast Analytics On Big Data
55mSort vs Hash A Duality
36mIn-memory data pipeline and warehouse at scale
28mBreakout Session
39mStorage-Partitioned Join for Apache Spark
48mContinuous Optimization for distributed BigData analysis
32mMastering stream-to-stream JOINs
47mJOINs Are Slowing You Down: Why Spark and Trino Won’t Replace Your DWH