Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)
summary
Instaclustr runs 797 Kafka clusters (3,363 brokers) with median size 3, mean 4.5 and max 96, and cluster sizes follow a Zipf-like long tail so larger clusters are likely to appear; total disk across clusters is about 5.6 PB disk. Top-10 clusters show 7–17,000 topics, ~22k–500k partitions, and a peak measured cluster of ~25M msgs/s biggest cluster with wide consumer latencies and heterogeneous hardware. Coarse per-broker metrics limit capacity planning; author recommends OpenTelemetry instrumentation plus benchmark-driven modeling for better predictions.
AI-written summary · about
chapters (7)
- 0:00Introduction to Kafka galaxies
- 3:38Kafka scalability and partitions
- 7:57Zipf’s law and cluster sizes
- 15:05Storage capacity analysis
- 17:32Top 10 clusters metrics
- 26:19Conclusions and future work
- 29:06Q&A
related talks
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
53mDemystifying Kafka
38mKafka Message Broker Under High Load
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
38mFault Tree Analysis Applied to Apache Kafka
41mApache Kafka simply explained
05mIntroduction to Kafka Internals
41mScaling Telemetry Systems with Streaming