Blink: Improvements to Flink & its application in Alibaba Search
summary
Blink is a Flink-compatible runtime that Alibaba deploys on a Hadoop cluster of more than 1,000 servers to serve mission-critical search and recommendation pipelines. It replaces Flink’s Yarn handling with per-job JobMaster for dynamic resource request/release, implements incremental checkpoints that upload only modified state to HDFS (RocksDB compaction periodically uploaded), adds pockets for elastic parallelism so operators change parallelism without losing state, and adds async ordered/unordered I/O, table API features (UDFs, joins, distinct counts) while keeping at-least-once / exactly-once guarantees and low latency.
AI-written summary · about
chapters (18)
- 0:00Introduction to Blink
- 1:10Why Alibaba uses Flink
- 3:20Search infrastructure challenges
- 6:53Stream-table duality explained
- 9:25String equivalence concept
- 11:34Blink’s business requirements
- 13:42Real-time A/B testing use case
- 14:46Online feature updates
- 16:31Why Flink over batch
- 17:57Starting the Blink project
- 19:22Table API improvements
- 20:46Runtime improvements overview
- 21:08New cluster architecture
- 23:01Dynamic parallelism
- 24:04Incremental checkpointing
- 25:30Async I/O feature
- 27:01Future challenges and work
- 28:26Community contributions
