The Magical Rebalance Protocol of Apache Kafka
summary
Kafka hides a generic coordination protocol that implements client-side partition assignment via a two-step join+sync with leader election, heartbeats and rebalances; it already supports surprising scale like 30,000 consumer groups and groups of hundreds. Concrete APIs use it: Consumer, Connect, Streams and Schema Registry, with hooks for custom assignors (for example sticky assignor) and a user metadata loophole that lets you assign non-partition tasks. The protocol works over the Kafka binary requests, has tunable timeouts, a generation ID for fencing and practical request size limits.
AI-written summary · about
chapters (24)
- 0:00Intro
- 3:18Partitions
- 4:12Replication
- 5:23Kafka Consumer
- 6:20Multiple Consumers
- 7:07Consumer Groups
- 10:00Requirements
- 13:17Whats a protocol to do?
- 13:53Layered protocol
- 15:00How the protocol is used today?
- 15:33Dramatis Personae
- 16:16Who is the Coordinator
- 18:01Joining Group
- 20:37Selecting an Assignment Protocol
- 22:42Kafka Protocol
- 23:04Sync Group
- 24:30Heartbeat
- 25:41Leave Group
- 26:08Rebalancing
- 29:39Schema Registry
- 32:31Stream-Stream Join
- 35:12To Summarize!
- 36:19Some future plans...
- 37:18References
related talks
16mAn introduction to Apache Kafka - this is Event Streaming, not just Messaging
47mIn the Land of the Sizing, the One-Partition Kafka Topic is King
16mIntroducing Apache Kafka
54mKafka: The Must Know Pub/Sub
33mKafka Microservices with Kafka plus Akka and Kafka Streams
01h 02mKafka for .NET Developers
56mBrick-by-brick: Exploring Apache Kafka®
59mBuilding Data Intensive Microservices with Apache Kafka