Writable Warm: Breaking the Read-Only Barrier in OpenSearch Tier...
summary
Writable Warm makes warm-tier indices writable by routing Lucene I/O through a composite directory that unifies local disk and remote blob stores (S3/GCS/Azure), serving only needed 8MB blocks via a ref-counted LRU cache and a block index input so reads fetch small chunks and writes upload segments back to remote. It uses a dummy switchable index input to evict full files, background staggered merges to reach 1–4 segments (avoiding costly force-merge spikes and cutting ~90% memory/GC overhead vs many segments), plus read-ahead prefetch and stored-field prefetch to mask remote latency.
AI-written summary · about
chapters (8)
- 0:00Writable warm introduction
- 1:25Storage tiering in OpenSearch
- 3:50Writable warm cost savings
- 4:56Writable warm architecture
- 11:38Optimizing warm tier search
- 18:25Writable warm use cases
- 20:52Key takeaways and plans
- 22:29Q&A
related talks
38mLight and Adaptive Indexing for Immutable Databases
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
31mMaximize Resource Efficiency with Separated Index and Search Workloads
20mBetter Indexing and Faster Retrievals for Decentralized Hot Storage
33mNext Gen of Datastores: Hot Queries, Cold Storage
46mSearch in the Cloud: separation of compute and storage
19mBuilding a Performant Configuration-Store Layer
01h 03mExtreme Performance Architecture