Metrics are Not Enough: Monitoring Apache Kafka
summary
Explains which concrete signals matter for Kafka operations and why: monitor under-replicated partitions as the primary early failure indicator, track the JMX metrics or a reporter for controller health and active controller count, alert on unclean leader election because it means data loss, run a simple end to end canary that produces and consumes a test message, audit by counting produced vs consumed messages, collect client metrics (producer retries, errors) and call producer.close on shutdown, and use tools like Confluent Control Center, flame graphs and thread sampling to find hotspots.
AI-written summary · about
chapters (12)
- 0:00Introduction to Kafka
- 1:04Monitoring Kafka basics
- 5:55Key performance metrics
- 9:02Building monitoring dashboards
- 11:53Under replicated partitions
- 16:13Cluster controller monitoring
- 19:06Broker health and restarts
- 22:12Client monitoring and audits
- 28:15Consumer performance
- 30:25Broker performance tuning
- 34:32Capacity planning
- 37:31Q&A and troubleshooting
related talks
32mDetect and Automatically Fix the Most Common Apache Kafka Problems
53mDemystifying Kafka
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
27mObserving Failure in Reactive Systems
05mIntroduction to Kafka Internals
38mStreaming Fast and Slow
38mKafka Message Broker Under High Load
38mFault Tree Analysis Applied to Apache Kafka