When your stream doesn't stream very well
summary
Apache Flink + Kafka jobs stall on test or batchy streams because windows wait for watermarks and sinks buffer small volumes; common failure modes are slow streams, slim (single-partition) streams, bursty/batchy streams and broken streams. Practical fixes: inject timestamp pings every ~0.5s into every Kafka partition (then de-duplicate after keyBy) or use the Flink 1.11 idle-source timeout with its tradeoffs, and set a periodic flush for the 2 MB buffered mutator in HBase so the last partial megabyte doesn't sit forever.
AI-written summary · about
chapters (10)
- 0:00Introduction to streaming issues
- 1:30Event time vs processing time
- 5:20Streaming windows and watermarks
- 10:02Problematic stream types
- 13:36Memory and bursty streams
- 16:48Solutions for slow streams
- 20:22Timestamp injection method
- 23:52Redefining topology concept
- 25:40Database buffering issues
- 27:29Final recommendations
related talks
38mTime-Based Aggregates: The Good, the Bad and the Ugly
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
38mInteractive Session: Optimising data streaming pipelines on Flink and Kafka
28mSweet Streams Are Made Of These
39mEvaluating Streaming Framework Performance for a Large-Scale...
40mRobust Stream Processing with Apache Flink
43mWatermarks - Practical Tool for Any Streaming and Event Processing Application
27mNavigating Streaming Infrastructure