OpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
summary
OpenSearch replaces local disk with cloud blob stores (S3/Azure/GCP) using segment-based replication so only primaries create segments, replicas fetch them from the remote store, cutting replica CPU and enabling compute/storage separation. Writes upload translogs to the blob store with a tunable 650 ms buffer to reduce small-file churn. Migration uses dual-replication (document + segment) per shard to convert live clusters with resumable progress; snapshots become near-instant for large fleets (hundreds of thousands of partitions).
AI-written summary · about
chapters (13)
- 0:00OpenSearch and Strangler Fig Pattern
- 1:03Software modernization approaches
- 3:11OpenSearch cluster architecture
- 6:21Disaggregated storage motivation
- 8:06Disaggregated storage principles
- 9:30Replication strategy seam
- 13:44Remote store integration seam
- 17:35Remote store integration challenges
- 22:29Cluster migration seam
- 26:43Key decisions and trade-offs
- 30:35Innovations and simplifications
- 32:23Key takeaways and best practices
- 33:48Q&A
related talks
46mSearch in the Cloud: separation of compute and storage
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
22mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
50mA Homegrown S3 Server: Challenges of Building Stable Storage
39mScaling OpenSearch: Next-Generation Shard Allo...
20mBuilding An Elastic Query Engine on Disaggregated Storage
43mDeep Dive Into OpenSearch Pull-Based Ingestion at Uber
35mSystems @Scale Summer Q&A Week 3