Observability-First Kafka: Engineering Visibility at Scale
summary
Shows concrete instrumentation: enable JMX, scrape Kafka with Prometheus JMX exporter, push client metrics via KIP-714 on Kafka 3.7+, and centralize logs as JSON to a collector like Filebeat or OpenTelemetry. Recommends Grafana and Grafana Loki for dashboards and logs. Calls out ~3500 JMX mbeans to cherry‑pick, emphasizes replication lag, network IO and error rates, and claims up to ~40% fewer incidents when MELT (metrics events logs traces) is fully applied.
AI-written summary · about
chapters (13)
- 0:00Introduction to Kafka Observability
- 0:49Why Kafka Needs Observability
- 2:11The MELT Stack Overview
- 2:37Deep Dive into Metrics
- 3:21Understanding Events in Kafka
- 4:31Importance of Logs
- 5:12Traces and Their Role
- 6:15Tool Recommendations
- 7:35Implementing Kafka Metrics
- 20:19Handling Kafka Logs
- 25:17Traces in Kafka
- 28:46Practical Implementation and Key Takeaways
- 30:29Helpful Resources and Conclusion
related talks
23mWhen Kafka Goes Cloud Native: Observability That Actually Works!
36mKafka Monitoring: What Matters!
06mIGNITE: Observability with Grafana & Prometheus for Kafka on Kubernetes (CFK)
38mObservability 101 with Spring and Micrometer
46mImplementing Observability in Java-Based Projects
32mUnderstanding Observability in Kubernetes
14mObserving the Future
55mBeginner's guide to observability