Kafka Internals I Wish I Knew Sooner
summary
Kafka reclaims disk only when segments roll so a one‑day retention with huge segment size effectively never frees space; default retention is seven days and the broker cleanable ratio is 0.5 so compaction/cleanup often waits. Idempotent producers are default since Kafka 3.0 and transactions require acks=all plus consumer isolation read_committed but exactly-once is fragile, lowers throughput, and still fails on consumer bugs. Rolling deploys plus cooperative sticky vs range/roundrobin rebalance choices can cause endless rebalance loops; start with defaults and measure.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:51Why Kafka internals matter
- 2:55How Kafka stores data
- 6:11Kafka as storage layer
- 9:07Replication and growth complexity
- 10:12Retention and compaction
- 13:45When data is deleted
- 14:07Delivery guarantees
- 15:56Exactly once delivery
- 19:26Consumers in transactions
- 22:58Consumer group coordination
- 24:21Rebalancing costs
- 26:07Rolling upgrade hell
- 27:53Avoid overengineering
- 29:46Q&A: Alternatives
- 32:01Q&A: Offset polling
- 33:12Q&A: Startup adoption
- 34:39Closing
related talks
36mKafka Without Disks: Making Streaming Brokers Disposable on K8S
02h 48mKafka: Map of Pitfalls for Developers and Ops
24mMake Your Kafka Cluster Production-Ready
50mKafka 4, fantastic?
35mKafka Deployment at Scale
48mKafka as a Message Queue: can you do it, and should you do it?
28mTackling Kafka, with a Small Team
04mPREVIEW: Running Kafka for Maximum Pain