The Cost of Apache Kafka
Talkat Data on Kubernetes Day NA 2025 · on Monday, 2025-11-10· at 15:25· Building B | Level 2 | B211-212·
summary
Walks through concrete cost drivers for Kafka on Kubernetes: brokers commonly run large VMs (examples: 64 GB RAM, 16 vCPU), storage often measures in terabytes and replication multiplies that cost (replication factor 3), and cross‑AZ traffic incurs cloud charges. Uses OpenCost for allocation and efficiency metrics and NetObserve (eBPF) to attribute cross‑zone network bytes into Prometheus so OpenCost can estimate network charges. Demonstrates a 3‑broker scenario with generated traffic, recommends tiered storage, zone‑aware consumer reads, and scheduled scale‑down (e.g., six→three brokers overnight) as cost levers.
AI-written summary · about
chapters (10)
- 0:00Apache Kafka costs
- 1:04Kafka cluster components
- 5:16Measuring costs with OpenCost
- 8:55Reducing storage costs
- 11:50Cutting network costs
- 16:26Offset commits and cross-zone traffic
- 17:31Scaling clusters by demand
- 19:17Adjusting replication
- 20:42Key takeaways
- 21:45Q&A
related talks
36mTaming the cost of Kafka workloads in the cloud
38mData Pipeline with Apache Kafka
36mKafka Without Disks: Making Streaming Brokers Disposable on K8S
38mKafka Message Broker Under High Load
32mShould you read Kafka as a stream or in batch? Should you even care?
25mOperating Cloud-Native Kafka
41mKafka as a Platform: The Ecosystem from the Ground Up
50mIntroduction to Apache Kafka