Deploying Kafka Streams Applications...

on YouTube ↗40m

" summary

Kafka Streams runs as partitioned tasks with a one-to-one task-to-partition mapping and local state backed by a compacted changelog topic, so scaling requires migrating those state shards. Kubernetes deploys these as containers; demo uses StatefulSet volumes, scales from two to three replicas, and recovers state from local disk or the changelog. Practical tips include using statefulset with anti-affinity, starting pods in parallel, setting a reasonable disruption budget, and reduce log segment size to cut recovery time for small states.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to Kafka Streams
2. 7:39Kafka Streams scaling deep dive
3. 22:04Deploying on Kubernetes basics
4. 32:01Stateful sets for Kafka Streams
5. 34:56Live demo walkthrough
6. 39:14Q&A and closing

" related talks

• 35mStreams Must Flow: Fault-tolerant streams processing apps on KubernetesVictor Gamov · Montréal Cloud Native Day 2019
• 30mDeploying SQL Stream Processing in Kubernetes with EaseAndrew Stevenson, Antonios Chalkiopoulos · KubeCon + CloudNativeCon Europe 2018
• 13mScaling Kafka on KubernetesShalini Katyayani Koney · Conf42 Kube Native 2025
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]Nenad Bogojevic · KubeCon + CloudNativeCon North America 2017
• 27mRunning Kafka in Kubernetes a practical guideKatherine Stanley · Voxxed Days Microservices 2018
• 41mApache Kafka + Apache Mesos = Highly Scalable Streaming MicroservicesKai Waehner · MesosCon Europe 2017
• 24mMake Your Kafka Cluster Production-ReadyJakub Scholz · Data on Kubernetes Day North America 2023