Swapping the Engine Mid-Flight: How We Moved Reddit’s Petabyte Scale Kafka Fleet to ...
summary
Reddit migrates roughly 500 brokers holding a couple petabytes and ~50M messages/sec from EC2 into Strimzi on Kubernetes with zero scheduled downtime and no data loss by running Strimzi brokers joined to the live cluster, adding plaintext nodePort listeners on 9095/9096, introducing a DNS bootstrap layer, and using Cruise Control (or scripted partition reassignment) to throttle replica and leadership moves. The process preserves reversibility, takes about 3–14 days per cluster, and finishes with a KRaft migration off ZooKeeper and back onto stock Strimzi.
AI-written summary · about
chapters (15)
- 0:00Live Kafka fleet migration intro
- 1:04Migration goals and constraints
- 3:09Hybrid Strimzi and EC2 brokers
- 5:35Hard-coded broker ID routing fixes
- 7:02Minimal Strimzi patches
- 9:27Strimzi brokers online safely
- 10:33Bulk data movement strategy
- 12:17Monitoring and throttling
- 14:42Consumer health and replication
- 15:47Zookeeper to KRaft transition
- 16:30Rolling back Strimzi config
- 17:34Key migration metrics
- 19:38Lessons learned
- 21:04Why Kafka replication wins
- 22:30Strimzi impact and Q&A
related talks
31mSimplifying Apache Kafka on Kubernetes With Strimzi
42mTransition to Apache Kafka on Kubernetes With Strimzi
33mManaging a Kafka cluster in KRaft mode with Strimzi
34mHow to Survive Managing Strimzi in an Enterprise
36mModernizing Nubank's Kafka Platform for the Next 10 Years Starting at...
29mMastering Kafka Workload Balancing With Strimzi’s Cruise Control Integration
32mStretching Strimzi: Exploring Multi-Cluster Kafka for High Availability
21mKeynote: Strimzi past, present and future