Running OpenSearch in the Real World: Migration, Mistakes and Maturity
summary
Runs OpenSearch on-prem on VMs as a Splunk replacement and then as an MDR logging platform ingesting 40GB to 1.2TB/day per customer, deployed with Terraform, Chef and Ansible. Uses Vector and Fluent Bit for ingestion, ISM for rollover at ~35GB and 7 days on disk, and on-prem S3 with searchable snapshots for long term storage. Builds an operator-like pipeline and opensearchcli to automate snapshot restores and deletions. Migration fail: accidental double ingestion, poor sizing and snapshot-induced OOMs force adding nodes and raising recovery throughput.
AI-written summary · about
chapters (13)
- 0:00OpenSearch migration journey
- 1:48Starting point and research
- 3:13Vector and Fluent Bit
- 4:16First cluster setup
- 5:17Operational service and safeguards
- 7:00Business simplification needs
- 7:42Cost savings with S3 snapshots
- 10:11Searchable snapshot operator
- 11:54Final architecture tiers
- 13:02Migration failure recovery
- 16:52Snapshot policy lessons
- 17:15Helpful tools and plans
- 19:23Q&A
related talks
45mOpenSearchCon Keynote Speakers
42mOpenSearch Migrations: Modernize Without Worry
10mFrom custom Solr to out-of-the-box OpenSearch
40mDTEX's OpenSearch Journey: From Migration to Innovation
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
24mClosing Remarks & Mastering Efficiency: Pathbreaking OpenSearch Index Management (Ingestion, Index)
40mInnovating in Open Source in Your Enterprise
14mKeynote: OpenSearch on Kubernetes at Atlassian