Digging Into OpenSearch’s Memory: Lucene and JVM Garbage Collection Under the...
summary
Indexing in OpenSearch generates a storm of short‑lived Lucene objects that drive frequent young GCs, high promotion rates and wasted heap; attaching visual VM + docker compose to a three‑node test cluster and running OpenSearch Benchmark reveals GC counts near 1,000 and object byte‑array lives dropping from ~500k to 15k when enabling UseStringDeduplication, cutting object heap from ~70% to under 50%. Practical levers shown: tweak G1UseAdaptiveIHOP and G1NewSizePercent/G1MaxNewSizePercent, split JVM/native memory ~50/50, and consider generational ZGC for very large heaps.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 1:25OpenSearch memory overview
- 2:08JVM heap vs native memory
- 3:32Lucene memory settings
- 4:54Lucene refresh vs flush
- 5:17Native memory and file cache
- 6:46Memory mapping and HNSW
- 7:51Inspecting memory mapping
- 8:56Monitoring clusters
- 9:59Lucene internals resources
- 10:21Lucene indexing and memory
- 11:47Reducing Lucene churn
- 12:32JVM heap patterns
- 13:14VisualVM with Docker
- 14:18OpenSearch test cluster
- 15:43OpenSearch Benchmark
- 16:49Preparing Benchmark tests
- 18:12Java GC basics
- 19:17G1 GC overview
- 20:27Generational garbage collection fundamentals
- 20:48Compressed pointers impact
- 22:17GC pauses trade-offs
- 23:25Object lifetimes hypothesis
- 24:11Adaptive IHOP in G1
- 25:35String duplication tuning
- 28:47Disabling adaptive IHOP
- 29:35OpenSearch G1 defaults
- 30:38Adaptive GC sizing
- 32:04OpenSearch 3.2 features
- 33:32ZGC in OpenSearch 3.2
- 34:35Project roadmap
- 35:16Q&A
related talks
39mAccelerating Aggregations with Lucene-Aware Execution in OpenSearch
49mTrash Talk: Exploring the JVM memory management
31mDemystifying Garbage Collection in Java
02h 41mSolving Memory Leaks in the JVM
52mGarbage collection for the humans
44mGarbage Collector: How to Tame the Beast and Make It Your Best Ally
51mJava Garbage Collection - The Basics
59mKeynote: Industry GC Insights from OpenJDK