15 Essential Do’s and Don’ts with Apache Kafka
summary
Fifteen concrete do’s and don’ts: use a replication factor 3 / min.insync.replicas=2 and acks=all; pick partition counts as a multiple of 6 after benchmarking per‑partition throughput; increase delivery.timeout.ms well beyond the 2‑minute retry window; enable producer idempotence and batch compression (Zstd, Kafka 3.8, level 1); register Avro/Protobuf schemas in a schema registry; route bad records to a DLT while copying offset/timestamp/headers; avoid dual‑writes to Postgres — use Debezium/Kafka Connect; outsource Kafka hosting.
AI-written summary · about
chapters (18)
- 0:00Introduction to Kafka talk
- 1:49Who uses Apache Kafka
- 4:01Kafka as a distributed system
- 8:00Naming conventions for topics
- 12:55Replication and partition strategies
- 17:14Upgrading Kafka clients
- 20:00Optimizing Kafka configurations
- 23:14Producer acknowledgments and safety
- 26:07Idempotence in Kafka
- 28:58Consumer group rebalancing
- 29:20Schema registry and data formats
- 32:37Handling data quality issues
- 38:03Key serialization across languages
- 40:13Using keys effectively in Kafka
- 42:02Read-after-write consistency
- 46:43Event-driven architecture best practices
- 48:50Topic retention and compaction
- 50:38Why not manage Kafka yourself
related talks
46mMake Your Kafka Applications More Robust
41mPatterns and anti-patterns for production ready Kafka Streams apps
50mIntroduction to Apache Kafka
46mBrick-by-Brick: Exploring the Elements of Apache Kafka®
39mKafka as a Platform: The Ecosystem from the Ground Up
29mGo Big with Apache Kafka
54mKafka: The Must Know Pub/Sub
41mKafka as a Platform: The Ecosystem from the Ground Up