Derived Source: Slash Storage Costs Without Losing Data in Open...
summary
Derived Source reconstructs the _source at fetch time from doc values and stored fields and is configurable via an index setting. It ships in OpenSearch 3.2 and cuts stored _source footprint in the example shown from 2076MB → 330MB (~60% savings), removes compress-and-decompress CPU during indexing, and shrinks merges. Supported types include booleans, numbers, dates, geopoints (not geo_shape), IP, keyword and more; extension hooks live in field mapper APIs. Expect near‑no regression for small hits but 10–100% latency regression on very large result sets and slower operation‑based recovery unless using translog workarounds.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:03OpenSearch storage internals
- 3:52Stored fields vs source field
- 7:24Source field storage cost
- 8:48Derived source concept
- 11:14Configuring derived source
- 16:36Implementation details
- 18:27Performance impact
- 22:45Limitations
- 27:23Future plans
- 29:10Q&A
related talks
41mConstant-Time Aggregations with Star-Tree in OpenSearch
40mLeveraging Lucene Skip Indexes for Faster Aggregations in OpenSearch
31mMaximize Resource Efficiency with Separated Index and Search Workloads
25mJourney To Cost-Optimized OpenSearch: A Deep Dive Into Segment Replication
39mAccelerating Aggregations with Lucene-Aware Execution in OpenSearch
36mSearch on the fly: how to lighten your Big Data
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...