Sharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
summary
OpenSearch adds an in-place shard split that turns a static shard count into a dynamic root→child hierarchy with mutually exclusive hash ranges and a logarithmic binary lookup for routing, implemented via local peer-recovery, hard-linked file copies and the existing shard-splitting Lucene predicate. The three-phase split (copy + translog retention, engine sync, quick cluster-state swap) delivers zero-downtime splits, optional force-merge to remove deleted docs, scales to 1,000+ shards in benchmarks, and is cloud-native and production-ready (targeting Q3).
AI-written summary · about
chapters (16)
- 0:00Shard splitting in OpenSearch
- 1:23Static shard count problems
- 3:08Hot shard challenges
- 4:10Existing solution limits
- 5:58New shard splitting requirements
- 8:26Shard splitting approaches
- 11:13Physical split mechanics
- 13:18Split workflow with peer recovery
- 17:31Activating child shards
- 19:13Shard splitting use cases
- 20:58Elastic scaling future
- 21:46Q&A
- 22:07Auto-scaling and shard limits
- 25:37Rollover vs physical split
- 28:26Dynamic splitting policies
- 30:11Summary
related talks
22mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
31mMaximize Resource Efficiency with Separated Index and Search Workloads
39mScaling OpenSearch: Next-Generation Shard Allo...
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...
46mMastering Snapshots in OpenSearch: From Basics To Advanced Strategi...
28mRunning OpenSearch at Scale in High-Traffic Gaming Systems
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
31mMigrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up