Lessons From Migrating To OpenSearch: Shard Design, Log Ingestion, and UI Decisions
summary
Migrated 150GB per day of logs from Elasticsearch to Amazon OpenSearch Service using Fluentd for shipping, ISM/aliases and rollover to cut ~3,000 shards down by creating 5‑day indices with primary shards=6 and 6-node cluster distribution, raising refresh to 120s and temporarily setting replicas=0 to drop CPU from bursts to ~30% average; later restored replicas=1 as investigation tooling and OpenSearch UI replaced Kibana, with remaining work on mapping, segment replication and S3 remote store.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:29Migration background
- 6:03Challenges encountered
- 7:06Cluster status issues
- 8:50CPU load imbalance
- 11:18Shard design strategy
- 14:22Performance tuning
- 16:54Red status incidents
- 19:21Field indexing optimization
- 22:52Future reliability strategies
- 23:56Lessons learned
- 24:17OpenSearch UI adoption
- 26:42OpenSearch UI challenges
- 28:05Migration outcomes
- 28:53Q&A
related talks
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...
28mRunning OpenSearch at Scale in High-Traffic Gaming Systems
04mOpenSearch Performance and Monitoring
19mEnhance Search Performance At Scale with
20mExtending OpenSearch Dashboards: Building a Unified Obser...
25mJourney To Cost-Optimized OpenSearch: A Deep Dive Into Segment Replication
32mUnleash Your Cluster's Potential with OpenSearch Benchmark