Beginners guide to balance your data across Apache Kafka partitions
summary
Explains how Kafka decides partition placement (explicit partition, key hashing, and sticky partitioner), shows that sticky reduces latency ~50% and CPU 5–15% in benchmarks, and gives a calculation: required partitions ≈ required consumer throughput ÷ single-consumer capacity (example 1 GB/s ÷ 100 MB/s → 10 partitions). Describes causes of hot partitions (bad keys, batching/linger misconfiguration), monitors like consumer lag and under‑replicated partitions, and fixes: add partitions, custom partitioner, or create/migrate to a new topic.
AI-written summary · about
chapters (7)
- 0:00Introduction to Kafka partitions
- 5:04Calculating partition count
- 13:15Default partitioner behavior
- 22:55Risks of unbalanced partitions
- 30:09Key selection for even distribution
- 37:46Monitoring and fixing skew
- 41:21Key takeaways and Q&A
related talks
01h 02mBeginners Guide to Balance Your Data Across Apache Kafka Partitions
36mApache Kafka Simply Explained
46mBrick-by-Brick: Exploring the Elements of Apache Kafka®
54mKafka: The Must Know Pub/Sub
32mShould you read Kafka as a stream or in batch? Should you even care?
16mAn introduction to Apache Kafka - this is Event Streaming, not just Messaging
38mInteractive Session: Optimising data streaming pipelines on Flink and Kafka
38mData Pipeline with Apache Kafka