The Lucene Scalpel: Tuning Low-Level Segments for Multi-Lingual Concurrency
summary
Shows concrete Lucene settings and measured gains: uses OpenSearch with per-field analyzers (SmartCN for Chinese, Arabic analyzer, English analyzers), adjusts tiered merge policy tuning, codec choice (LZ4 vs best_compression), RAM buffer, refresh and force-merge timing. Claims: mixed-language P95s were ~580ms and Arabic ~340ms vs English 45ms; after tuning P99 latency drops about 62%, indexing throughput rises ~3x, and segment count falls >41% on the same hardware. Configuration snippets and metric comparisons are provided.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:46Lucene segments multilingual challenges
- 3:11Segments shards inverted index
- 4:15Tokenization language complexities
- 6:22Performance disparities languages queries
- 7:25Merge pressure tiered policy
- 8:08Six tuning areas concurrency
- 9:52Query strategies real-world results
- 10:56Language mix tuning
- 12:01Production results improvements
- 13:04Code examples tuning
- 14:47Monitoring segments refresh
- 15:50Global product strategy
- 16:53Morphology differences
- 17:56Percentiles versus averages
- 18:37Q&A
related talks
37mFast Decompression Lucene Codec #bbuzz
37mShipping Lucene 10.0, 25 years in the making
39mDigging Into OpenSearch’s Memory: Lucene and JVM Garbage Collection Under the...
29mSolr Sparse Faceting #bbuzz
43mEnhancing Query Performance Through Intelligent Data Co-location
41mSearching large data sets in (near) constant time
41mE-Commerce search at scale on Apache Lucene
40mApache Lucene 6: What's coming next? #bbuzz