Processing Millions of Events per Second with Apache Kafka on Kubernetes
summary
Unity runs Kafka on Kubernetes with the Strimzi operator, scaling clusters up to 60 brokers and handling ~20 million events/sec across 22 clusters (peak ~17 GB/s). Observability uses Prometheus dashboards and alerts, latency is measured with KMinion, and Cruise Control rebalances clusters. Tiered storage offloads to GCS (5 PB remote, ~84 TB local) and cuts costs about 65%. Configuration uses KCL templating, topics use three replicas with min.insync.replicas=2, and the setup proves production-ready.
AI-written summary · about
chapters (24)
- 0:00Kafka on Kubernetes intro
- 3:20Unity’s Kafka pipeline
- 7:44Kafka architecture basics
- 9:53Kafka clusters at Unity
- 12:19Tiered storage savings
- 13:0020M events per second
- 14:25Tiered storage in practice
- 16:55Kafka vs Netflix scale
- 17:40Monitoring with Red Panda
- 18:43Why multiple clusters
- 20:06Strimzi operator benefits
- 21:51Strimzi cluster definition
- 22:32Cluster management with KCL/Argo
- 25:21Rack awareness setup
- 26:24Schema changes handling
- 28:12Partitioning for throughput
- 31:47Workload Identity auth
- 32:52Observability setup
- 33:13Key Kafka alerts
- 37:10Consumer lag monitoring
- 38:11Cluster latency with Kminion
- 38:52Broker rebalancing
- 41:01Cruise Control enablement
- 41:45Kafka cluster lifecycle
related talks
13mScaling Kafka on Kubernetes
26mProduction Grade Kafka on K8s
31mEvent driven communication with Kafka on Kubernetes
30mFrom Running to Operating: Real-World Strimzi in Production
31mSimplifying Apache Kafka on Kubernetes With Strimzi
42mTransition to Apache Kafka on Kubernetes With Strimzi
24mMake Your Kafka Cluster Production-Ready
34mHow to Survive Managing Strimzi in an Enterprise