How LinkedIn is leveraging kafka for log aggregation and monitoring
summary
Describes how LinkedIn uses Apache Kafka as a distributed commit log for log aggregation and service monitoring, integrating Kafka with the ELK stack for metrics and intra-application queuing. Covers practical use of Kafka and ELK in production operations and managing ELK clusters for multiple teams.
AI-written summary · about
chapters (9)
- 0:00Start
- 3:02Objectives
- 5:07Solution
- 5:42What is kafka?
- 7:07Kafka with Consumer Groups (Queuing)
- 9:47Why Kafka
- 11:42How does it work
- 13:42ELK @ LI
- 17:12Use Cases
related talks
35mApache Kafka as a Monitoring Data Pipeline
39mKafka as a Platform: The Ecosystem from the Ground Up
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
41mKafka as a Platform: The Ecosystem from the Ground Up
29mApache Kafka and the Next 700 Stream Processing Systems
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
34mChasing the Stream Processing Utopia
33mObservability-First Kafka: Engineering Visibility at Scale