Streaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs
summary
Stripe builds a custom dual source that backfills from S3 and transparently hands off to Kafka, but large backfills stall due to watermark skew and ballooning buffered state. Diagnosis shows uneven key‑group assignment and oversized S3 segments; enabling flash SSD optimize gives ~2x peak throughput, while using Flink’s global aggregate manager to compute a global min watermark and deliberately sleep to induce event‑time backpressure restores lockstep progress. Result: reliable horizontal scalability and much faster completed backfills.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:53Dual source challenges
- 5:47Watermark skew mechanics
- 8:18Parallelism imbalance cause
- 10:24Throttling fast tasks
- 12:55Performance results
- 13:58Key lessons learned
related talks
43mA Tale of Dual Sources: Pictures of Grief and The Job Manager’s Clock
38mStreaming Fast and Slow
27mCustomizing exactly-once processing at scale with Flink
38mFaster and Furiouser: Flink Drift
26mHow Streaming Helps Your Staging Environment and Sandboxes Stay Up To Date
42mEverything is connected: How watermarking, scaling, and exactly once impact one another in Pravega
42mApache Flink Worst Practices
32mBig Data Processing : Where do we go next ?