Practical Migration From Distributed OpenSearch Environments To Enterprise-wide SIEM Platform
summary
Migration consolidates multi-account logs into a shared OpenSearch SIEM using S3 replication + Lambda triggers, snapshot sync and Python + AWS4Auth, converting formats and field types (text→ip) for CIDR queries. 102.2 GiB migrated in one month including 230,000 Apache and 1.4M IIS entries, using 10k-record bulk batches and Lambda chunk_size tuning. Cold/snapshot restore handles historical data (≈13 TB), warm-indexes block multi-index snapshots so individual snapshots are used, and OR1 instances plus index/retention changes yield 25% cost reduction ($1,900/month).
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:26Project background
- 3:14Existing environment
- 4:17Integration architecture
- 5:20Instance changes
- 6:27Migration with snapshots
- 7:29Cost reduction results
- 8:10Migration troubleshooting
- 9:13Operational efficiency
related talks
41mScaling Welfare With Open Source: How NAV Migrated To Managed OpenSearch
33mSecurity Logging Use Cases: Building an Open-Source SIEM
22mFireside Chat with AWS
32mMove to managed analytics
20mMigrating a Large Scale Search Dataset in Production in a Highly Available...
20mSIEM with Elastic – Insights for Security, Freedom for DevOp
16mA Thrilling Journey into Data-Driven Security Automation
48mThe Journey of a strongly coupled Monolith into the Cloud Land