Kafka: Map of Pitfalls for Developers and Ops
summary
Covers common Kafka failure modes and operational fixes: explains how producer batching and blocking futures (due to metadata, serialization and partitioning differences like Java sticky partitioning) can surprise developers, and how delivery.timeout plus acks (0/1/all) affect durability vs latency. Demonstrates tooling and practices including local testing with toxicproxy and kafka-containers, Strimzi on Kubernetes, Cruise Control for rebalancing, service binding for injecting configs, and discusses static membership, incremental cooperative rebalances and how disks, ISR or min.insync.replicas can break a cluster.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 2:27Kafka basics for devs
- 15:31Java message demo
- 20:09Cluster setup
- 30:05Metadata and partitions
- 41:21Updating Kafka in prod
- 1:03:26Message consumption and offsets
- 1:18:23Health monitoring
- 1:32:05Event-driven architectures
- 1:43:57Q&A
- 1:53:58Alerts and ops procedures
- 2:03:04Consumer rebalancing
- 2:19:21Replicas and availability
- 2:33:12Kafka security
- 2:41:41Reactive streaming demo
- 2:46:38Conclusion
related talks
40mKafka, DevOps...And Resilience for All
04mPREVIEW: Running Kafka for Maximum Pain
01h 02mKafka for .NET Developers
54mKafka: The Must Know Pub/Sub
01h 00mKafka for .NET Developers
35mKafka Internals I Wish I Knew Sooner
51mFive years of highs and lows with Kafka
38mA Tale of a Meshi Kafka: Securing Kafka Deployment When Istio Is Used