What is Behind Alibaba's Double 11 Shopping Festival
summary
Flink (Blink improvements) powers Tmall’s Double 11 dashboards, processing peaks of about 4 billion records/sec and ~7 TB/s with end-to-end latency under one second; the deployment runs ~35,000 real-time pipelines and cost roughly 1.5 million CPU units. Engineering delivers stream-batch unification by unifying SQL/Table and DataStream APIs, watermark semantics, unified stateful operators, pluggable shuffle (task-managed, YARN native, remote shuffle), and pipeline-region scheduling. Results: exactly-once semantics, 4–10x faster development, half the batch resources, and batch runtimes cut by ~4 hours.
AI-written summary · about
chapters (20)
- 0:00Flink and Double 11 intro
- 1:05Double 11 Shopping Festival
- 2:09Double 11 growth & Flink's role
- 3:56Flink evolution at Alibaba
- 5:23Stream-batch unification in Flink
- 7:08Defining stream-batch unification
- 8:16Making stream processing first-class
- 9:20Goals of stream-batch unification
- 10:04Stream vs batch differences
- 11:26Unified Flink APIs
- 12:09Time & watermarks in Flink
- 13:37Stateful ops: stream vs batch
- 15:02Data transfer in Flink
- 16:45Unified shuffle services
- 17:06Scheduling & failure handling
- 18:30Pipeline regions for execution
- 19:13User experience in unification
- 19:56Flink future milestones
- 20:37Double 11 results with Flink
- 22:03Key takeaways
related talks
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
27mLarge Scale Stream Processing with Blink SQL at Alibaba
47mBring Flink to Large Scale Production
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
25mData: Moving with the flow
20mData Technology Enabled AI Solutions
43mLeverage the power and simplicity of SQL on Apache Flink
45mApache Flink — Real-Time Data Processing