Scaling Kafka on Kubernetes

on YouTube ↗13m

" summary

Claims sustained 100,000 messages per second with <5 ms P99 latency and ~99.9% availability using Kubernetes primitives: Kafka as StatefulSets, SSD-backed PVs, XFS with tuned mount options, CPU requests of 2–4 CPU cores per broker and 8–16 GB heap, HPA driven by consumer lag and partition throughput, MirrorMaker for cross-region replication, GitOps CI/CD for automated rollouts and rollbacks, and Prometheus/Grafana monitoring; rolling upgrades, autoscaling and disaster recovery are demonstrated end to end in production.

AI-written summary · about

" chapters (15)

1. 0:00Introduction to Kafka on Kubernetes
2. 0:32Understanding the Problem Space
3. 1:11Why Choose Kafka on Kubernetes?
4. 2:06Key Pillars of Kafka on Kubernetes
5. 3:08Resource Allocation Strategies
6. 4:06Optimizing Storage for Kafka
7. 5:13Enhancing Network Performance
8. 6:02Auto Scaling Kafka Clusters
9. 7:00Multi-Cluster Deployments
10. 7:50Operational Excellence with GitOps
11. 8:35Monitoring and Metrics
12. 9:24Performance Metrics and Achievements
13. 10:28Disaster Recovery Strategies
14. 11:38Conclusion and Key Takeaways
15. 12:39Q&A and Further Discussion

" related talks

• 42mProcessing Millions of Events per Second with Apache Kafka on KubernetesRasmus Selsmark · DevOps Pro Europe 2026
• 35mKafka Deployment at ScaleOleksandr Nitavskyi · Highload fwdays'19
• 24mMake Your Kafka Cluster Production-ReadyJakub Scholz · Data on Kubernetes Day North America 2023
• 31mEvent driven communication with Kafka on KubernetesRaihan Khan · KCD Dhaka 2024
• 26mProduction Grade Kafka on K8sAnand Iyer · Kubernetes Community Days Chennai 2022
• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 47mRunning Kafka in Kubernetes: a practical guideKatherine Stanley · Devoxx Belgium 2019
• 27mRunning Kafka in Kubernetes a practical guideKatherine Stanley · Voxxed Days Microservices 2018