Efficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
summary
Flink writes one data file per writer per checkpoint and frequent commits produce thousands of small files and metadata entries, hurting read speed; adding a compaction job that runs as a Flink post‑commit topology rewrites data and metadata and uses existing cluster resources. A demo shows 300 generated records becoming ~800 write results and a monitor that emits 10 checks in 10 minutes and triggers 4 compactions, where compaction reduced 10,000 files to 80. New pieces include syn V2 API, monitor source and trigger manager to schedule compactions.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 4:00Iceberg table mechanics
- 10:00Flink Iceberg sync challenges
- 16:27Proposed solution and API
- 18:15Demo and results
- 23:40Summary and future work
related talks
19mStreaming Event-Time Partitioning With Apache Flink and Apache Iceberg
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
45mApache Flink — Real-Time Data Processing
47mBring Flink to Large Scale Production
36mFlink SQL: Unified Streaming and batch for hive integration
36mDynamically Generated Flink Jobs at Scale
46mFlink SQL in 2020: Time to show off!
38mApache Flink: A Very Quick Guide