Make Your Kafka Applications More Robust
summary
Practical advice to make Kafka-based apps more resilient: use Dead Letter Queues that keep original topic/partition/offset/timestamp and add a readable failure reason; validate or reject bad payloads at the producer via a schema registry (Avro/Proto/JSON Schema) instead of letting consumers fail. Operational recommendations include using replication factor ≈3, avoiding acks=0, and exercising producer config, idempotence and retries in integration tests. Also presents a Kafka proxy that rewrites metadata, enforces rules, adds encryption, offloads large payloads to S3, and enables chaos testing, with working demos.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:31Common Kafka problems
- 6:28Kafka project maturity
- 7:31CI/CD limits
- 10:21Config errors
- 13:53Poison pills solutions
- 19:36Dead letter queues
- 25:15Schema registry in practice
- 33:00Advanced Kafka proxy
- 44:39Conclusion & Q&A
related talks
41mPatterns and anti-patterns for production ready Kafka Streams apps
52m15 Essential Do’s and Don’ts with Apache Kafka
39mKafka as a Platform: The Ecosystem from the Ground Up
41mKafka as a Platform: the Ecosystem from the Ground Up
41mKafka as a Platform: The Ecosystem from the Ground Up
19mHow to Fail Your Event-Driven System
38mA Crash Course in Error Handling for Streaming Data Pipeline
54mKafka: The Must Know Pub/Sub