Centralized Log Management
summary
A production report on a centralized log-management stack using Logstash to collect logs, Elasticsearch for storage and search, and Kibana for visualization. It covers throughput (about 18,000 entries in 15 minutes on a single node), scaling by adding nodes, performance issues when many indices slow searches, and operational techniques: grok filters for normalization, TLS/certificates for transport, and Curator/scripts for retention. It also notes practical limits: lack of fine-grained access control (need Shield/Enterprise) and spotty documentation, and emphasizes planning indexing, retention, and access control.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:10What are logs
- 4:37Decentralized log problems
- 7:27Open-source log approaches
- 10:45Elastic Stack and Kibana
- 12:52Elastic Stack in practice
- 18:16Security and performance
- 20:45Critical points and improvements
- 22:38Conclusion
- 23:19Q&A
related talks
14mCentralized Logs using ELK (Elasticsearch, Logstash, Kibana) Stack
42mMake your logs work for you
37m(Finally) Sufficient Monitoring and Logging
43mPractical Application Logging and Monitoring
01h 00mApplication Logging & Logstash
53mtake care of your logs
01h 02mCentralised Logging without the blood, sweat and tears
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution