Vendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
summary
OpenSearch adds a pluggable composite engine that natively queries open columnar files (tested with Parquet) using Apache Arrow in-memory layout and DataFusion as the SIMD-capable query engine. Benchmarks show ingestion rising from ~50k to ~55k docs/sec with derived source, Parquet giving ~2.5x throughput and 87% smaller storage, and Arrow stream ingestion hitting 6x throughput via Arrow streams. Dual-mode uses Parquet primary + Lucene secondary for fast text plus analytics and yields ~58% faster ClickBench aggregations with extensibility for Lance, ORC, Iceberg committers.
AI-written summary · about
chapters (8)
- 0:00OpenSearch analytics evolution
- 2:05OpenSearch Lucene limits
- 4:51Composite engine architecture
- 10:25Indexing changes and settings
- 13:14Ingestion performance improvements
- 15:21Search performance with DataFusion
- 16:03Future plans and community
- 16:45Q&A on formats and vector search
related talks
45mAccelerating OpenSearch With Streaming: Apache Arrow, Flight, Data...
26mColumnar Storage: Redefining Data Management for the Modern Era
42mUsing OpenSearch as a Database With DataFusion
42mColumnar Storage: Redefining Data Management for the Modern Era
46mEfficient Data formats for Analytics with Parquet and Arrow
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
37mFrom Inverted Index to Columnar Vectorized Execution Search
22mAccelerating Analytics with Apache Arrow