Centralized logging and sleeping well at night
summary
Describes building a geo-distributed centralized logging system on OpenSearch storing 300+ TB and serving 3,000 users. Covers how the architecture and tech stack evolved as load grew, and lists key lessons about what the team would do differently if starting from scratch.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:22Old architecture problems
- 7:12New system concept
- 10:19Nightmares and fixes
- 13:10Log storage and lifecycle
- 16:04Quota system importance
- 23:30Summary and advice
- 45:55Q&A
related talks
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
50mAn Elasticsearch Cluster Over 200 TB
25mOne Size Does Not Fit All: A Polystore Architecture for Logs and Traces
20mHow We Implemented a Near Real-Time Logging Solution Using Fluent Bit And OpenSearch
30mCentralized Log Management
37m(Finally) Sufficient Monitoring and Logging
18mFrom Logs To Signals: Rethinking System Monitoring With OpenSearch
28mFrom Fragmented Logs to Unified Insights: Scaling O11y