Taming the cost of Kafka workloads in the cloud
summary
Cloud Kafka workloads show network as the biggest surprise: a simple app at 100 MB/s in/out can incur >$3,000/month just for cross‑AZ traffic. Enable follower fetching to read from local replicas and reduce consumer cross‑AZ egress to near zero, use producer compression factor ~3x (lz4 default) to cut network and storage at the cost of CPU, and autoscale consumers with kada scaling to zero to avoid idle compute charges.
AI-written summary · about
chapters (10)
- 0:00Introduction and agenda
- 1:24Cloud computing cost surprises
- 5:41Cost drivers for Kafka workloads
- 8:30Network cost example breakdown
- 10:42Follower fetching technique
- 16:47Compression for Kafka records
- 23:23Lag-based consumer scaling
- 28:43Scaling consumers to zero
- 30:10Summary and key takeaways
- 31:38Q&A session
related talks
24mThe Cost of Apache Kafka
25mOperating Cloud-Native Kafka
36mKafka Without Disks: Making Streaming Brokers Disposable on K8S
33mUncover Kubernetes cost-optimization trends
33mKubernetes cost optimization - How to actually do it
45mCloud Computing: Cloud in a cost-effective way
43mKubernetes cost optimisation - How to actually do it
14mCut Costs, Not Performance: Implement Kubernetes Cost Optimization Strategies