Patterns and anti-patterns for production ready Kafka Streams apps
summary
Covers concrete, actionable patterns: use the Topology Test Driver and Testcontainers for unit and integration tests; prefer mapValues/flatMapValues so key stays stable and avoid extra repartition subtopologies; store state locally in RocksDB with Kafka changelogs and enable standby replicas to cut failover time. Recommends Avro/Protobuf, tune batch.size and linger.ms, pick compression (lz4/snappy), set num.stream.threads, monitor JMX metrics, name processors, handle deserialization/production errors, and add a shutdown hook.
AI-written summary · about
chapters (18)
- 0:00Introduction to Kafka Streams
- 1:03What is Kafka Streams
- 2:47Kafka Streams architecture
- 4:52Streams and state stores
- 8:20Operations in Kafka Streams
- 10:05Word count example
- 12:57Kafka Streams tasks and partitions
- 17:30Production readiness dimensions
- 18:34Testing Kafka Streams apps
- 20:40Scaling Kafka Streams
- 24:09Producer and consumer tuning
- 25:55Stateful app optimizations
- 27:20Monitoring and observability
- 28:46Naming processing steps
- 30:09Ensuring high availability
- 33:44Exception handling
- 35:11Shutdown hooks and cleanup
- 35:31Q&A session
related talks
46mMake Your Kafka Applications More Robust
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
01h 00mIn Depth Kafka Streams
03mInterview
50mApplication development and data in the emerging world ...
40mEvent Streaming Applications with Kafka Streams, Spring Kafka and Actuator
27mNavigating Streaming Infrastructure
38mA Crash Course in Error Handling for Streaming Data Pipeline