Introduction to Kafka Internals

on YouTube ↗05m

" summary

Kafka acts as a distributed, partitioned, replicated commit log: brokers hold partitions, producers write messages, consumers in a group claim partitions. Frequent consumer rebalances cause cascading lag and outages when members join or drop; avoid by thoughtful partitioning and preventing long-running consumer work. Configure timeouts: set producer delivery.timeout.ms greater than broker recovery time and tune session.timeout.ms and heartbeat.interval.ms to reduce unnecessary rebalances and keep throughput stable.

AI-written summary · about

" related talks

• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 01h 31mDeveloping Real-Time Data Pipelines with Apache KafkaJoe Stein · SpringOne2GX 2015
• 38mKafka Message Broker Under High LoadArtem Vybornov · РИТ++ 2017
• 16mIntroducing Apache KafkaKatherine Stanley · Devoxx Belgium 2019
• 54mWhen Everything Went Kafka: Apache Kafka and the ConsumerGrigoriy Koshelev · JPoint 2023
• 50mIntroduction to Apache KafkaJames Ward · Devoxx US 2017
• 42mA Kafka Client’s Request: There and Back AgainDanica Fine · Berlin Buzzwords 2023
• 41mApache Kafka simply explainedOlena Kutsenko · Berlin Buzzwords 2022