Time-Based Aggregates: The Good, the Bad and the Ugly
summary
Time-based aggregations on Flink break when Kafka partitions are skewed or replayed and watermarks lag, causing operators to stall or drop late events; details include how watermark merging uses the latest input watermark, how out-of-order timestamps and late arrivals interact with window closes, and how excessive state growth or full checkpoints (disk/space limits, stalled Kafka consumer offsets) lead to jobs that stop making progress. Concrete fixes: tune partitioning, watermark strategy, and checkpoint cadence.
AI-written summary · about
chapters (5)
- 0:00Introduction to time-based aggregates
- 7:59Problems with time-based operations
- 15:00Watermarks and out-of-order data
- 23:08Production issues and debugging
- 35:09Solutions and best practices
related talks
29mWhen your stream doesn't stream very well
38mInteractive Session: Optimising data streaming pipelines on Flink and Kafka
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
40mRobust Stream Processing with Apache Flink
45mData Stream Processing Concepts and Implementations
39mEvaluating Streaming Framework Performance for a Large-Scale...
38mEventually, time will kill your data pipeline
19mStreaming Event-Time Partitioning With Apache Flink and Apache Iceberg