PREVIEW: Multi-Tenant, Multi-Cluster, Hierarchical Messaging
summary
Netflix runs a hierarchical, multi-cluster Kafka service that handles over a trillion messages per day by allocating topics to shared or dedicated clusters based on SLA and migrating them when needed. Infrastructure routers stitch clusters together, enhanced client libraries and proxies provide abstracted access, and management/self‑service tools plus monitoring automate operations. Designed to avoid monolithic-cluster pain—rolling restarts and image updates that take days for hundreds of brokers—the system implements transparent Kafka failover and cross-cluster routing so clients see no topology changes.
AI-written summary · about
chapters (3)
related talks
42mRunning a Massively Parallel Self-Serve Distributed Data System at Scale
36mModernizing Nubank's Kafka Platform for the Next 10 Years Starting at...
46mCassandra Serving Netflix @ Scale
31mEvent driven communication with Kafka on Kubernetes
33mMicroservices, Events, and Breaking the Data Monolith with Kafka
42mPartial Multi-Tenancy on Kafka Using Strimzi at LittleHorse
11mScaling Kafka in the Cloud
24mAdvanced RabbitMQ Strategies: A case study