Future of Lucene's MMapDirectory: Why use it & what's coming with Java 16 & later?
summary
Discusses Lucene's use of MMapDirectory for disk access to avoid user‑space copies and use the OS page cache, and explains current 32‑bit‑style addressing limits that force fallbacks and extra copies as indexes grow into gigabytes. Describes using Java’s Foreign Memory Access API (Panama, JEPs 370/383/393) available in Java 16+—memory segments and direct access—and a draft Lucene implementation that promises copy‑free reads and support for indexes in the tens or hundreds of terabytes, with work ongoing to integrate this into Solr and Elasticsearch.
AI-written summary · about
chapters (8)
- 0:00Introduction to talk
- 2:40Memory mapping basics
- 6:53Java memory management
- 11:16Virtual address space
- 17:36Lucene and memory mapping
- 24:23Java 16 and later changes
- 30:55Future of MMapDirectory
- 34:37Q&A and summary
related talks
53mThe future of Lucene's MMapDirectory
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
43mCaching and Beyond - Smarter Data Processing with Java
34mProject Panama in Action: Building a File System
37mFast Decompression Lucene Codec #bbuzz
51mIn-Memory Computing - The Big Picture
01mIn-Memory Computing — The Big Picture
40mTrash Talk - Exploring the Memory Management in the JVM