Kafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replication

on YouTube ↗28m

" summary

kafka-mirror-tester runs two Kubernetes clusters (US-East and EU-West), spins up ~30 Kafka brokers per cluster on 40–50 node clusters, and uses Golang producers and consumers to validate cross‑Atlantic replication by load, latency and correctness. Tests include broker kill, cluster resize, adding topics, increasing partitions (replicator fails to detect partition count change and requires topic recreate), and 10% simulated packet loss; replication with uReplicator keeps data durable and catches up, monitored with Prometheus & Grafana.

AI-written summary · about

" chapters (14)

1. 0:00Introduction and motivation
2. 3:58Jepsen-inspired testing approach
3. 5:25Kafka replication basics
4. 7:35Test suite design and goals
5. 9:00Kubernetes setup overview
6. 11:11Cluster architecture and components
7. 14:50Broker failure test case
8. 16:43Cluster resize test case
9. 17:47Replicator scaling test case
10. 19:14New topic replication test
11. 20:18Partition scaling test case
12. 21:23Packet loss simulation test
13. 22:49Go producer and consumer code
14. 27:09Limitations and future work

" related talks

• 47mRunning Kafka in Kubernetes: a practical guideKatherine Stanley · Devoxx Belgium 2019
• 29mKafka on Kubernetes: Does it really have to be the hard way?Victor Gamov · Montréal Cloud Native Day 2019
• 26mProduction Grade Kafka on K8sAnand Iyer · Kubernetes Community Days Chennai 2022
• 31mEvent driven communication with Kafka on KubernetesRaihan Khan · KCD Dhaka 2024
• 13mScaling Kafka on KubernetesShalini Katyayani Koney · Conf42 Kube Native 2025
• 24mMake Your Kafka Cluster Production-ReadyJakub Scholz · Data on Kubernetes Day North America 2023
• 33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]Nenad Bogojevic · KubeCon + CloudNativeCon North America 2017
• 44mTaming Apache Kafka Using GitOpsNatale Vinto · JCON WORLD ONLINE 2023