From Bottlenecks To Breakthroughs: Optimizing Aggregation in OpenSearch for Lightning-Fas...
summary
Original monthly date-histogram that took ~7–8s drops to ~162ms when rewritten as per-month range queries and to ~2s when run serially, achieved by leveraging Lucene’s points/BKD index and a count API; a new multi-range traversal reduces minutely/daily/hourly latencies 2–3x and handles high bucket cardinality; using a skip index plus bitset popcount yields ~3x speedups for filtered aggregations; overall claims include up to 100x improvements for some histograms and contributions merged back into Lucene.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:05OpenSearch aggregations
- 3:33Bottlenecks discovery
- 5:19Default execution flow
- 7:29First optimizations
- 9:36Lucene's count API
- 12:07BKD trees for range queries
- 15:16Filter rewrite improvements
- 16:23Filter rewrite limits
- 17:51Multi-range traversal
- 21:11Multi-range limits
- 22:13Skip index optimization
- 24:21Skip indexes with doc IDs
- 26:49Vectorized execution
- 27:53Star Tree index
- 28:37Q&A
related talks
37mFrom Inverted Index to Columnar Vectorized Execution Search
39mReal Time Datamining and Aggregation at Scale
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
21mReporting Systems Built on Top of Hadoop & Lucene
28mA complete Tweet index on Apache Lucene #bbuzz
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
26mHigh-Performance Full-Text Search on Apache Iceberg Using Lucene-Integrate...
33mMake Your Data FABulous