A Kafka Producer’s Request: Or, There and Back Again
summary
Follows a producer.send() from client to disk and back: keys serialize to bytes, default partitioning uses sticky partitioning (not round-robin) across six partitions, topic replication factor is three, batching defaults to ~batch size 16KB and linger.ms 0 with buffer.memory 32MB on the client, max.request.size ≈1MB, acks defaults to all, max.in.flight.requests is 5, num.network.threads 3 and io threads 8, replication fetchers poll (~500ms nominal, often 5–8ms), and metrics like request latency, localTimeMs and remoteTimeMs pinpoint where delays occur.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 2:18Getting data into Kafka
- 3:14Creating a producer client
- 3:33Data is in Costco
- 4:13What happens when you issue a producer request
- 5:15How to write data into Kafka
- 6:22Partitioner class
- 10:35Batching
- 11:26Batch Size
- 12:46Linger Ms
- 13:47Buffer Memory
- 15:10Metrics
- 17:17Compression
- 19:05Additional Configurations
- 23:27Request Metrics
- 24:37Nested Objects
- 25:59Leaving Application Developer Domain
- 26:41Socket Receive Buffer
- 27:27Processing the Request
- 28:37Request Queue
- 29:36Request Handler
- 30:34Write to Disk
- 31:54Kafka Configurations
- 32:41Kafka Monitoring
- 33:21Tiered Storage
- 34:36Purgatory
- 37:20Response Queue
- 37:52Back to Network Threads
- 38:33Summary Metrics
- 39:38Conclusion
related talks
42mA Kafka Client’s Request: There and Back Again
05mIntroduction to Kafka Internals
53mDemystifying Kafka
16mIntroducing Apache Kafka
44mFive Years of the Good and the Not-So-Good with Kafka
48mKafka as a Message Queue: can you do it, and should you do it?
30mWhy Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)
41mApache Kafka simply explained