A Tale of Dual Sources: Pictures of Grief and The Job Manager’s Clock
summary
Stripe builds a custom Flink dual source that bootstraps from S3 archives and then hands over to Kafka, because Kafka retention is only 3 days retention while archive processing can take a week. The solution enumerates and partitions S3 files, stores progress and max Kafka offsets in operator state, and forces a coordinated handover by waiting for a checkpoint then triggering a job restart (checkpoint+restart). It works in production but causes watermark skew and state buffering; mitigation includes resubmitting to recalculate end time and tuning parallelism.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:09Motivation for deduplication
- 7:33Challenges with existing approaches
- 10:25Deciding to build a dual source
- 12:13Basics of Flink sources
- 12:33First hurdle wrapping S3 source
- 16:45Second hurdle Kafka retention
- 20:14Third hurdle wrapping Kafka source
- 27:55Final dual source solution
- 32:55Lessons learned
- 35:26Future of Flink sources
- 35:52Q&A
related talks
15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs
34mA journey of scaling high-volume low-latency stateful processing
27mCustomizing exactly-once processing at scale with Flink
28mIt's Time To Stop Using Lambda Architecture
43mDynamic Scaling - How Apache Flink adapts to changing workloads
31mTale of Stateful Stream to Stream Processing
26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
37mBuilding and operating large-scale streaming applications at King