Beyond Hot Spots: Revolutionising OpenSearch Performance Through Heat-Aware Operations
summary
Proposes a heat-aware allocator that scores each shard as a five-dimensional vector (CPU, JVM, disk I/O, disk usage, OS requests), sampled every 5s and aggregated into 15m buckets (96 samples → 384 bytes per shard; 10k shards ≈ 3.58 MB). New allocation uses power-of-two random choices among allocation-decider–eligible nodes picking the cooler node; rebalancing runs when node heat > mean + 2σ and moves small, high-heat, low-write shards. Adds heat watermarks, a metrics API/dashboards, and a path to predictive allocation or one-shot LP optimization.
AI-written summary · about
chapters (17)
- 0:00Heat-aware shard allocation intro
- 1:06Current shard allocation in OpenSearch
- 7:04Hotspots from homogeneous nodes
- 8:10Segment replication impact
- 10:14Limitations of allocation strategies
- 11:18Weight function problems
- 13:49Manual hotspot mitigation
- 14:33Shard heat as metric
- 19:50Heat-based shard placement
- 21:15Rebalancing with heat thresholds
- 21:56Event-based rebalancing
- 24:04Allocator architecture options
- 28:17Linear programming for allocation
- 29:19Heat-based allocator benefits
- 30:45Heat-aware features roadmap
- 32:29Key takeaways and future work
- 33:12Q&A
related talks
44mBringing Down the Cost of Your OpenSearch Clusters
28mRunning OpenSearch at Scale in High-Traffic Gaming Systems
45mMonitoring Your OpenSearch Cluster Effectively
31mMaximize Resource Efficiency with Separated Index and Search Workloads
24mHotRing: A Hotspot-Aware In-Memory Key-Value Store
43mMulti-tenancy for all workloads
41mOptimising Apache Spark and SQL for Improved Performance
15mOptimizing Query Performance in Big Data Systems