Migrating from Self-Managed ElasticSearch to OpenSearch
summary
Migrates a 50TB Elasticsearch 7.16 cluster to Amazon OpenSearch with zero downtime by running a full POC that holds all data in a staging cluster, then backfills updates and flips traffic. Uses 50 TB / 60 billion docs migrated via Logstash per-index with 60 instances, tuning input slices, batch sizes, refresh/replica settings and memory (up to 45 GB) to avoid overload. Validates with Kafka-driven ingest replay and k6 API load tests against targets of 3k/11k writes per second and 1k/5k reads, reaches production using scaled node types and reports an 11% cost decrease.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:30About Yotpo
- 1:55Customer Data Platform CDP
- 2:45YPO CDP
- 3:42Targeted profiles campaigns
- 5:06Cluster overview
- 5:46What were the pains
- 8:04Why AWS
- 9:17Challenges
- 10:19Zero Downtime
- 11:09Functionality
- 13:26Performance
- 14:46Why Migration
- 15:44Migration Options
- 16:48Avoiding Production Overload
- 18:35Execution
- 20:27Pipeline
- 24:12Results
- 24:29Cost
- 25:34Migration
- 30:03Since then
- 30:57Next steps
related talks
31mMigrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
42mOpenSearch Migrations: Modernize Without Worry
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
28mOperational analytics with AWS
21mNoOps for OpenSearch: The Future of Self-Healing Search Infras...
35mInnovating at Scale: Lessons from Pegasystems' OpenSearch Journey
38mFree enterprise search with OpenSearch