Modernizing Nubank's Kafka Platform for the Next 10 Years Starting at...
summary
Nubank moves from raw EC2 Kafka to Strimzi on Kubernetes, running multi-account VPC peering and Transit Gateway to retain control and compliance while handling 1 trillion messages per month across ~150 clusters and 160,000 topics. A cell-based architecture plus a topic-management service (France), CLI (nucle), Cruise Control and a concurrent-cluster live-migration pattern lets services consume from both clusters, swap producers, then cut consumers. Targeting 100 topics/day to finish migration in under a year; clusters now create in one hour and 15 of 153 are live on the new infra.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:09Ten years of Kafka at Nubank
- 8:04Legacy Kafka challenges
- 12:23Self-hosted Kafka on Kubernetes
- 15:39New Kafka platform design
- 19:33Rolling out modern Kafka
- 23:06Concurrent clusters migration
- 29:06Batch migration results
- 31:59Future plans
- 35:13Q&A
related talks
32mNubank's experience scaling microservices operations
30mRunning multiple Nubanks at once
27mSwapping the Engine Mid-Flight: How We Moved Reddit’s Petabyte Scale Kafka Fleet to ...
30mFrom Running to Operating: Real-World Strimzi in Production
42mProcessing Millions of Events per Second with Apache Kafka on Kubernetes
26mProduction Grade Kafka on K8s
31mSimplifying Apache Kafka on Kubernetes With Strimzi
05mPREVIEW: Multi-Tenant, Multi-Cluster, Hierarchical Messaging