Dynamically Generated Flink Jobs at Scale
summary
A metadata-driven Flink pipeline generates about 120K jobs/day to ingest, validate and milestone 90K datasets across a 700 TB lake handling ~7 trillion records daily. Replacing Spark/MapReduce yields a 3–5x latency reduction and lower CPU/memory; serialization relies on Kryo caches for evolving schemas and wide rows. Large DAGs are split into subjobs, coGroup is accelerated with bloom filters, and heuristics plus slot caps limit noisy tenants; example jobs range from seconds for millions of rows to ~60 minutes for 300 GB jobs.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:08Traditional Data Warehouse
- 8:55Flink Architecture
- 12:00Metadata
- 13:05Job Graph
- 14:22Job Graphs
- 15:16Sub Graphs
- 16:03Serialization
- 17:59Code grouping
- 18:46Demo
- 19:51Changing Name
- 20:31Removing Records
- 21:18Using Bloom Filters
- 22:14Bloom Filter Performance
- 22:59Tuning Cluster Management
- 27:42Performance Examples
- 30:17Performance Comparison
related talks
45mApache Flink — Real-Time Data Processing
31mFlink @WORKDAY: scaling of ingestion data pipelines to S3
47mBring Flink to Large Scale Production
36mSources, Sinks, and Operators: A Performance Deep Dive
25mData: Moving with the flow
24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
21mFlink's application at Didi
43mLarge Scale User Behavior Analytics by Flink