Accelerating OpenSearch With Streaming: Apache Arrow, Flight, Data...
summary
A new streaming framework integrates Apache Arrow with Flight and gRPC and a DataFusion-capable query layer to stream per-segment Arrow batches to the coordinator, drop global ordinals, remove shard-level priority queues, and do continuous merges. Results show about 2× throughput and up to 8× lower latency (example 2.5s → 230ms), ~50% lower data‑node CPU, and heap for big arrays shrinking roughly 4GB → ~200MB; feature lands as an opt‑in in 3.2.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:03OpenSearch query engine evolution
- 3:53Why streaming aggregation
- 5:18Benefits of streaming term aggregation
- 6:43Streaming joins and analytics use cases
- 11:00Apache Arrow columnar format
- 13:48Apache Arrow Flight data transfer
- 16:13OpenSearch aggregation basics
- 20:45Term aggregation bottlenecks
- 27:05Streaming aggregations solve bottlenecks
- 34:26Future of columnar formats
- 38:19Next steps for streaming
- 41:28Q&A
related talks
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
01h 02mApache Arrow: High-Performance Columnar Data Framework
33mArrow in Flight - New Developments in Data Connectivity
32mVectorized Query Processing for CPUs using Apache Arrow
45mApache Arrow: In Pursuit of Speed
58mComposable Data Systems, the new frontier
22mAccelerating Analytics with Apache Arrow
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.