From Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
summary
Calura migrates observability to OpenSearch using AWS managed OpenSearch with Data Prepper for ingest, Fluent Bit + Prometheus exporters and OpenTelemetry for traces, handling 8 TB/day ingest and spikes near 6 GB/s. Design uses NVMe nodes, single shards with zero replicas where acceptable, ISM lifecycle to move data to UltraWarm and cold tiers, and smart retention giving 30+ day retention for key logs, cutting costs up to 60%.
AI-written summary · about
chapters (10)
- 0:00Observability cost dilemma
- 3:36SAS CloudWatch vs DIY options
- 8:51Managed OpenSearch for scalability
- 13:11Ingest storage and retention strategy
- 20:46Self-healing observability system
- 24:40Security and multi-signal setup
- 27:50Architecture overview and data flow
- 29:37Cost savings and user feedback
- 31:22Key takeaways and summary
- 33:29Q&A on retention and Netflix use case
related talks
14mKey Considerations For Success When Deploying A Solution For Best-of-breed Observability At Scale
28mOperational analytics with AWS
24mFinOps for Observability: Optimizing OpenSearch Costs Without Sacrificing Visibility
41mScaling Welfare With Open Source: How NAV Migrated To Managed OpenSearch
38mMigrating from Self-Managed ElasticSearch to OpenSearch
46mOperational Logs and Event Management for HPC S...
25mUnifying Observability: Weaving Prometheus, Jaeger, and Open Source Together to Win
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...