Beyond the Watermark: On-Demand Backfilling in Flink
summary
Uber runs Flink to compute driver incentives but watermarks are global so retroactive incentives break streaming. The solution creates an on-demand source that, when an incentive registers, scans the driver-status change log (Kafka-like) and emits incentive-specific records into the same job, preserving checkpoints and exactly-once semantics. This lets one single pipeline for thousands of incentives replace thousands of jobs, produces progressive or final payout records, handles backfills, and runs in production with acceptable resource use.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 1:21Driver incentive use case
- 4:00Original pipeline architecture
- 7:36Challenges with scaling
- 10:04Addressing retroactive data
- 12:54The on-demand source solution
- 17:14Templated pipeline approach
- 21:17Feature requests for Flink
- 27:10Q&A session
related talks
20mFlinking, Fast and Slow
45mThe State of Flink and how to adopt Stream Processing
38mFrom Flink 1.3 to 1.4
40mTowards Flink 2.0: Unified Batch & Stream Processing
27mBatch/Stream Ask-Me-Anything
38mBridging the Chasm Between Research & Software Development
47mAll the DataOps, all the paradigms
36mTime-based domain events - how to deal with time