Kafka Without Disks: Making Streaming Brokers Disposable on K8S
summary
A migration from MSK to AutoMQ moves durability into object storage durability and turns brokers into stateless brokers (no replication, no PVs), tested on clusters with roughly 5 brokers, 5,000 partitions and ~2 MB/s and a larger CDC cluster with ~300k partitions and 3‑day retention. Operational wins: brokers restart in ~30s and costs fall about 2.5×. Tradeoffs: cold‑cache penalties show as cache hit ratios and S3 latency percentiles; schema registry and linking required nontrivial recovery work.
AI-written summary · about
chapters (19)
- 0:00Introduction to Kafka without disks
- 1:07Kafka architecture and disk dependency
- 3:18Problems with current Kafka setups
- 6:07Decoupling storage and compute in Kafka
- 7:09Tiered storage approaches in Kafka community
- 10:20Slack's tiered storage proposal
- 11:02Bad approach with batch coordinator
- 12:26KIP-932: Disclos 2 architecture
- 14:50AutoMQ: Practical decoupled Kafka
- 17:20AutoMQ primitives and elastic log
- 20:53Role of cache in AutoMQ
- 21:35Choosing different wall types
- 23:40Migration to AutoMQ at Fresh
- 25:24CDC warehouse cluster results
- 26:52Migration challenges and lessons
- 28:18Outbox cluster with EFS wall
- 29:42New metrics and operational model
- 31:06Takeaways and future of Kafka
- 31:47Q&A session
related talks
24mThe Cost of Apache Kafka
13mScaling Kafka on Kubernetes
35mKafka Internals I Wish I Knew Sooner
24mMake Your Kafka Cluster Production-Ready
26mProduction Grade Kafka on K8s
37mDisaster Recovery Options running Apache Kafka in Kubernetes
28mTackling Kafka, with a Small Team
29mKafka on Kubernetes: Does it really have to be the hard way?