Deep dive into an Elasticsearch plugin for query-time joins
summary
Siren Federate implements query-time joins on Elasticsearch by hash-partitioning documents across nodes, building an in-memory hash table per shard and returning parent-doc subsets as Lucene bitsets. Benchmarks on clusters (12–36 data nodes, up to 15B docs) show joining ~140M docs per side finishes ~1.9s; scaling to 1B and 2B per side yields ~5s and ~7.6s. Performance improves ~2x after switching from row reads to vectorized columnar processing (Apache Arrow) and benefits from a bitset cache and a pipeline of batch operators.
AI-written summary · about
chapters (10)
- 0:00Introduction to the platform
- 0:40The case for Siren Federate
- 3:15Basic concepts of joins
- 4:26Query capabilities
- 12:16Technical implementation
- 15:51System internals
- 20:15Data processing pipeline
- 24:13Join execution steps
- 26:46Benchmarking and performance
- 30:21Future work and Q&A
related talks
38mAccelerating Query Performance in CrateDB
21mJoining in Lucene
39mStorage-Partitioned Join for Apache Spark
41mParallel and distributed big joins in H2O
18mSpeed Up Your Search: Concurrency and Merge Policies to the Rescue
01h 00m50 Shades of Join in Apache Lucene
42mUsing OpenSearch as a Database With DataFusion
16mSo many results... Give them to me!