Building Self-Balancing Monitoring of Kafka Topic Processing Using Kafka Itself

on YouTube ↗46m

" summary

Describes building a self-balancing monitoring pipeline for Kafka topics using Kafka itself—specifically Kafka Streams, Kafka Queryable State stores, and Quarkus. The team avoided heavy frameworks like Spark or Flink, adapted Kafka Streams to minimize internal system topics while retaining resilience and scalability, showing that 1:N topic joins and monitoring tasks can be handled efficiently with a lightweight Kafka Streams-based solution.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 1:48Monitoring task description
3. 4:39Kafka standard metrics issues
4. 8:15Freshness lag as key metric
5. 11:24Monitoring restart problems
6. 15:20Optimization via compact topics
7. 22:50Partition distribution issues
8. 27:03Custom Partition Assigner
9. 31:25Reliability and scalability
10. 34:54Monitoring and Kafka Streams conclusions
11. 38:26Q&A

" related talks

• 50mSpring Kafka beyond the basicsTim van Baarsen · J-Fall 2022
• 01h 44mData processing with Kafka Streams and Spring Framework.Dmitry Sergeev · Devoxx Ukraine 2020
• 01h 29mEvent stream processing using Kafka StreamsFredrik Vraalsen · Berlin Buzzwords 2018
• 27mStream processing with Kafka streamsHamza Belmellouki · BlaBlaConf 2022
• 40mEvent Streaming Applications with Kafka Streams, Spring Kafka and ActuatorKo Turk · Voxxed Days Bucharest 2022
• 01h 11mBootiful Stream Processing!Viktor Gamov · Java Z-Day 2021
• 48mSpring for Apache Kafka the advanced featuresTim van Baarsen, Kosta Chuturkov · Spring I/O 2025
• 46mKubernetes. From 0 to Production-Grade with Java.Kevin Dubois · Voxxed Days CERN 2025