In the Land of the Sizing, the One-Partition Kafka Topic is King
summary
Claims a concrete sizing method that starts by measuring a one-partition baseline with the Kafka perf tools shipped in the distribution bin folder then applies the formula partitions = max(target_write / measured_write, target_read / measured_read) so a 40k target with measured write ~20,500 rps and read ~37,400 rps yields two partitions. Demonstrates using kafka-reassign-partitions to rebalance brokers, shows replication increases network and CPU load so choose replication factor carefully and warns about range vs cooperative assignor causing consumer pauses.
AI-written summary · about
chapters (7)
- 0:00Start
- 10:45How Do You Do Partitioning into Kafka
- 15:37How Hadoop Cluster Used To Work
- 22:46Kafka Reassign Partitions
- 27:40The Event Pool
- 38:37Throughput
- 41:45Unit of Durability
