Using OpenSearch as a Database With DataFusion
summary
Integrates DataFusion with OpenSearch to run SQL-style planning, joins, and aggregations by implementing a DataFusion table provider that executes shard queries via JNI and returns zero-copy Arrow vectors. Execution pushes down projection and predicate push-down into Lucene, collects docID batches, performs docvalue lookups into Arrow vectors, and streams results over Flight for local mini-joins on worker coordinators. Early experiments on a three-node cluster show 30–35% latency reduction on a terms aggregation and notably lower heap usage.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:28Apache Arrow benefits
- 7:28Apache DataFusion
- 9:56OpenSearch aggregations today
- 15:54Aggregation challenges
- 16:17DataFusion with OpenSearch
- 19:11Early performance results
- 20:34OpenSearch join use cases
- 26:56Future architectures
- 28:41Implementation challenges
- 35:13Q&A
related talks
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
05mKeynote: OpenSearch: From Code to Community
09mIntroducing Opensearch in existing multisource enterprise App
06mSponsored Keynote: Foundation-Led Innovation: OpenSearch's Impact on Modern Data I...
55mVector Search Made Simple: Getting Started with OpenSearch for AI Applications
26mOpenSearch and the Path to a Trillion-s...
41mDynamoDB, Neptune and OpenSearch: A Unified Approach to Collaborative Editing and Search #JSConfAM25