Support Tiered Storage in Strimzi Operated Kafka
summary
Integration of KIP‑405 tiered storage into Strimzi operated Kafka with a custom S3 plugin moves cold segments to S3, enabling longer retention and cheaper EBS use while keeping recent data on local disks. Key fixes include tuning max fetch wait, separating upload and fetch thread pools, and log segment sizing. Optimizations (multi‑part upload and concurrent S3 range fetch) yield roughly 5x throughput; tests on six brokers show about 270 MB/s per‑broker for a single consumer and drop to ~20–30 MB/s with many topics. Proposal lands in Strimzi post‑0.40 for native config support.
AI-written summary · about
chapters (13)
- 0:00Tiered storage in Strimzi Kafka
- 1:26Streaming pipeline architecture
- 3:55Kafka cluster setup at Apple
- 5:26Cost challenges driving tiered storage
- 7:13Integrating tiered storage in Kafka
- 10:03Integration technical challenges
- 13:19Upgrading Kafka to 3.6
- 14:43Tuning remote storage performance
- 16:33Optimizing S3 plugin library
- 18:44Benchmarking tiered storage
- 21:39Performance results and bottlenecks
- 23:46Strimzi open source contributions
- 26:44Q&A
related talks
31mSimplifying Apache Kafka on Kubernetes With Strimzi
32mStretching Strimzi: Exploring Multi-Cluster Kafka for High Availability
30mFrom Running to Operating: Real-World Strimzi in Production
27mSwapping the Engine Mid-Flight: How We Moved Reddit’s Petabyte Scale Kafka Fleet to ...
21mKeynote: Strimzi past, present and future
42mProcessing Millions of Events per Second with Apache Kafka on Kubernetes
24mMake Your Kafka Cluster Production-Ready
26mProduction Grade Kafka on K8s