Scaling Log Management: PB-Scale Migration From Elasticsearch To...
summary
Migrates a PB-scale log platform from Elasticsearch 6.8.4 to OpenSearch 2.x using dual writes and backfill from Kinesis, storing ~8 PB with 21-day retention and ingesting ~5M events/sec; adopts searchable snapshots on S3 to separate compute and storage, uses subclustering (node attributes) to isolate tenants, upgrades hot hosts to i4 instances, and reduces nodes from ~1,500 to ~1,000; benchmarks and custom index-state scripts validate read/write performance and deliver a 30% cost reduction.
AI-written summary · about
chapters (5)
- 0:00Log management at Freshworks
- 8:46Migrating from Elasticsearch to OpenSearch
- 16:50Migration strategies and zero-downtime
- 23:30Q&A on cluster management
- 35:32Final thoughts
related talks
38mMigrating from Self-Managed ElasticSearch to OpenSearch
35mInnovating at Scale: Lessons from Pegasystems' OpenSearch Journey
41mScaling Observability: From Chaos to Clarity with Open Source Tools
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
31mMigrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
48mBuilding Petabyte-Scale PostgreSQL Deployments
29mHow (not) to Scale Elasticsearch for Data Analytics!
42mOpenSearch Migrations: Modernize Without Worry