Flink @WORKDAY: scaling of ingestion data pipelines to S3
summary
Flink jobs scale from 4k+ buckets and tiny per-bucket streams to single-bucket batch loads of 150M+ messages by replacing the default sink with a two-phase S3 sink that removes a 3s per-message existence check, keeps in-flight data in state with a size threshold, switches renames to an AWS S3 client and parallelizes commits to restore throughput and exactly-once. Job completion for 12-hour batches is signaled only after checkpoint-confirmed commits via a secondary stats flow. Testing uses a message firehose, local Kafka and a performance-evaluator subproject.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:40Multi-tenancy
- 3:13Security requirement
- 4:312019 - ingestion challenges
- 7:55Sink before the performance improvements
- 8:51First attempt - windowing
- 9:50Second attempt: Streaming Sink
- 11:14Final (and Successful!) attempt: new sink
- 14:05Keying and partitioning
- 18:03Introducing a secondary observability flow
- 18:39Extract metadata for new flow
- 20:53report a notification AFTER data is stored
- 24:46Logic to report ingestion state
- 29:01Appendix2: How to handle state breaking change of the sink (without breaking it)
related talks
36mDynamically Generated Flink Jobs at Scale
46mFlink SQL in 2020: Time to show off!
47mBring Flink to Large Scale Production
43mA Data Streaming Architecture with Apache Flink #bbuzz
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
45mApache Flink — Real-Time Data Processing
21mFlink's application at Didi
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond