Unspoken overhead of the Inverted Index
summary
Explains why the inverted index is built per text field, how Lucene segments merge, and why bulk inserts help; shows analysis steps with the term vectors API and how analyzers produce tokens that become terms; demonstrates a JVM stack trace generating 56 terms that mostly never get searched; highlights choices — use keyword fields, disable doc values when not aggregating, or pick match_only_text to drop positions — and reports community benchmarks of roughly ~10% storage savings for match_only_text while acknowledging query and memory tradeoffs.
AI-written summary · about
chapters (12)
- 0:00Inverted index basics
- 1:27Apache Lucene in OpenSearch
- 3:35Reversed search logic
- 5:19Building index segments
- 9:38Text analysis process
- 14:10Bloated term storage
- 17:02Optimizing index structure
- 20:16Keyword vs doc values trade-offs
- 22:24Hidden index data
- 24:30Match-only text indexing
- 25:56Index mechanics recap
- 26:21Q&A
related talks
26mFrom __icontains to search
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
26mPhysical vs. Logical Indexing with IDEA: Inverted Deduplication-Aware Index
37mAlgorithms & data-structures that power Lucene & ElasticSearch
25mApache Lucene: From Text Indexing to Artificial Intelligence
55mFull-Text Search Demystified
40mLeveraging Lucene Skip Indexes for Faster Aggregations in OpenSearch
46mFull-Text Search Internals with Elasticsearch