Make Your Kafka Cluster Production-Ready
Talkat Data on Kubernetes Day North America 2023 · on Monday, 2023-11-06· at 15:15· W471ab, McCormick Place·
summary
Kubernetes-hosted Kafka needs explicit choices: use block or local persistent storage not NFS, enable TLS/mTLS and ACLs, and integrate Strimzi’s user operator for auth. Set CPU and JVM memory requests/limits carefully—set requests and limits equal or avoid memory limits—and reserve RAM for OS page cache. Coordinate rolling updates to minimize controller moves and wait for replicas to resync before restarting next broker. Spread brokers across zones with pod topology spread constraints, monitor consumer lag, and use Cruise Control for rebalancing and plan backups.
AI-written summary · about
chapters (14)
- 0:00Production-ready Kafka clusters
- 1:46Infrastructure requirements
- 3:09Resource requests for brokers
- 5:39Security best practices
- 8:07Monitoring Kafka
- 9:30Availability and reliability
- 12:21Rolling updates and controllers
- 13:47Kubernetes zones and broker distribution
- 15:35Disruptions and probes
- 17:00Disaster recovery and backups
- 17:41Performance tuning and JVM
- 18:25Noisy neighbors in Kubernetes
- 19:26Cluster balancing
- 21:13Summary and resources
related talks
26mProduction Grade Kafka on K8s
33mManaging a Kafka cluster in KRaft mode with Strimzi
47mRunning Kafka in Kubernetes: a practical guide
31mSimplifying Apache Kafka on Kubernetes With Strimzi
29mKafka on Kubernetes: Does it really have to be the hard way?
29mMastering Kafka Workload Balancing With Strimzi’s Cruise Control Integration
37mDisaster Recovery Options running Apache Kafka in Kubernetes
33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]