7 Petabytes of Logs in Elastic. How We Did It
summary
Tinkoff describes scaling their in-house monitoring platform Sage, which uses Elasticsearch to collect logs and metrics from thousands of services. The talk covers handling growth to 7 petabytes of logs across 15 clusters (370 nodes) and the operational challenges and solutions they applied to manage that scale.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 2:16Building our own system
- 7:23How Elasticsearch works
- 10:13Large storage principles
- 11:58Data upload with Kafka
- 13:24Searching across data centers
- 15:15Small clusters benefits
- 16:59Node roles and setup
- 18:24Quotas and limits
- 22:59Logstash replacement
- 26:52Index lifecycle
- 31:06Custom index features
- 34:18Java optimizations
- 39:16Advice and future plans
- 45:19Stricter data schemes
- 47:26Blocking bad queries
- 48:51Handling log backlog
- 50:35Custom log database
- 53:02Monitoring and security
related talks
50mAn Elasticsearch Cluster Over 200 TB
43mCattle vs. Pets - A Cost-Effective Elasticsearch Architecture to Scale-Out...
35mHow to Efficiently Manage Logs in Large-Scale Kubernetes Clusters
29mHow (not) to Scale Elasticsearch for Data Analytics!
48mEvolution of the Metrics Pipeline: How the Architecture Scaled with Load
40mLarge-Scale Logging Made Easy
30mData @Scale Seattle- Sundaram Narayanan
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...