Sharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
summary
OpenSearch implements in-place physical shard splitting that replaces a hot root shard with multiple child shards using root-child shard model and mutually exclusive hash ranges, avoiding reindexing and the 2.1 billion-doc limit. Lookups use a logarithmic playoff lookup (binary search over sorted ranges) so queries scale O(log n). Child creation uses local peer-style recovery with hard-linked local recovery, translog retention and replay, brief shard-permit handoff for zero write-downtime; benchmarks show rollover can cause ~200% search latency.
AI-written summary · about
chapters (6)
- 0:00Shard splitting intro
- 3:31Hot shard limits
- 6:18Hot shard approaches
- 8:44In-place shard splitting
- 16:26Use cases and extensions
- 18:11Q&A
related talks
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
43mMulti-tenancy for all workloads
23mSharding for hyperscale and geographically distributed relational databases
32mSharding for mission-critical, internet-scale apps
40mNon-Obvious Things to Know When Coding in OpenSearch
31mUber’s Innovations in OpenSearch for a Cloud Native, Serverles...