Running Kafka in Kubernetes: a practical guide
summary
Shows a tested example Kafka cluster on Kubernetes using three brokers with replication factor 3 and min.insync.replicas=2, demonstrating leader elections when brokers are killed and how producers fail when ISR falls below the minimum. Recommends Kubernetes primitives: use statefulset + PVC (not Deployment) to avoid broker ID conflicts, use headless service + advertised.listeners for direct broker routing, monitor JMX metrics for readiness, run Zookeeper quorum checks, and consider Helm or the Strimzi operator to automate installs and rolling updates.
AI-written summary · about
chapters (14)
- 0:00Start
- 2:53Rise of containers
- 5:33Data centric
- 7:38Apache Kalka is an open source, distributed streaming platform
- 8:51Topics
- 9:50Partitions
- 10:24Replication
- 13:59Kubernetes health checks
- 20:56Rolling updates
- 28:00Zookeeper health checks
- 38:33Headless services
- 39:46Listeners and Advertised Listeners
- 43:08Automation
- 45:05Kubernetes Operators
related talks
27mRunning Kafka in Kubernetes a practical guide
26mProduction Grade Kafka on K8s
24mMake Your Kafka Cluster Production-Ready
29mKafka on Kubernetes: Does it really have to be the hard way?
33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]
37mDisaster Recovery Options running Apache Kafka in Kubernetes
33mManaging a Kafka cluster in KRaft mode with Strimzi
13mScaling Kafka on Kubernetes