Understanding Bottlenecks & Conquering them
summary
Describes concrete changes to remove cluster-manager and shard bottlenecks: publish cluster state to a remote store so each node downloads needed pieces instead of the master pushing 10MB to 2,000 nodes, batch shard allocation and introduce logical parts/pods to limit placement complexity, add a shard balancer that uses CPU/memory/disk/IOPS “heat” and an online shard split API to split hot shards without downtime, and harden admin APIs with pagination, cancellation, precomputed node stats and admission control.
AI-written summary · about
chapters (17)
- 0:00OpenSearch scaling limits
- 1:04Speakers and roles
- 1:24OpenSearch cluster structure
- 3:31Why infinite scaling fails
- 7:00Cluster manager single point of failure
- 8:05Scalable cluster manager
- 13:06Remote state cluster publication
- 16:34Shard allocation bottlenecks
- 19:43Logical partitions for shards
- 20:45Heat-based shard balancing
- 21:50Online shard splitting
- 23:15Request routing optimization
- 24:39Partitions for request flow
- 25:45Admin APIs at scale challenges
- 27:31Precomputing stats for aggregation
- 28:56Future snapshot storage optimizations
- 29:38Reader-writer separation
related talks
39mScaling OpenSearch: Next-Generation Shard Allo...
45mSharding and Things We'd Like To See Improved
31mBeyond the Limits: Scaling Kubernetes Controllers Horizontally
37mSharding: Growing Systems from Node-scale to Planet-scale
25mAvoiding Overload in Distributed Systems
17mBreaking GitOps Bottlenecks: The Power of Sharding
31mMove Less, Move Faster: Speeding Up Citus Cluster Scaling
24mWhen and How Should you Shard?