Migrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
summary
Migration from self‑managed Elasticsearch 7.x to Amazon OpenSearch Service and OpenSearch 2.3 uses restore from snapshot, rolling parallel clusters and direct ingestion from Kafka, cutting peak search latency from ~1000 ms to under 200 ms and CPU from ~90% to ~40% at 15M concurrency. Tooling includes Kafka Connect and Flink, NVMe SSDs, static index templates, 10–50 GB shard sizing, more replicas, thread‑pool and refresh tuning, and an ML‑driven autoscaler; dual clusters run for a week before final cutover.
AI-written summary · about
chapters (13)
- 0:00Self-managed Elasticsearch migration
- 1:03Migrate to Amazon OpenSearch
- 3:11Migration roadmap and execution
- 5:17Migration patterns overview
- 7:46Tuning and cutover best practices
- 10:12Dream11’s migration journey
- 10:56Dream11’s platform pain points
- 14:28Legacy Elasticsearch challenges
- 17:55Dream11’s migration plan
- 20:46Performance tuning at Dream11
- 25:43Cost optimization and scaling
- 26:46Final OpenSearch migration
- 28:36Results and future plans
related talks
38mMigrating from Self-Managed ElasticSearch to OpenSearch
38mFree enterprise search with OpenSearch
35mInnovating at Scale: Lessons from Pegasystems' OpenSearch Journey
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...
40mOperating Opensearch - The Kubernetes Way
28mThe top five things to know about OpenSearch
28mRunning OpenSearch at Scale in High-Traffic Gaming Systems