Machine learning with Flink in Weibo

on YouTube ↗18m

" summary

Weibo uses Flink to unify batch and stream pipelines for recommendation: a single codebase replaces separate Storm/Spark/MapReduce paths, with online feature joining (example 10 minute join window), immediate flush-on-join and sample composition to resend late positives. The platform splits PS clusters for online prediction and training, stores snapshots in HDFS/RocksDB/Gemini, and claims support for 100 billion parameters at 1.1 million QPS with ~10 minute iteration cycles.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 0:44Brief introduction
3. 1:29Architecture
4. 3:00Development
5. 4:13Models
6. 4:44Infrastructure
7. 8:15Sample Generating Process
8. 8:49Multistream Training
9. 13:00Job Management
10. 14:32Multimedia Future Generation
11. 16:14Better Flink
12. 17:20Conclusion

" related talks

• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 20mUsing Flink for Versatile Feature Engineering PipelinesMichal Ciesielczyk · Flink Forward Global Virtual 2020
• 25mSharing what we love: The Apache Flink story at PinterestChen Qin · Flink Forward Global 2021
• 15mFeature Stores: Powering Machine Learning Infrastructure with Apache FlinkSimba Khadder · Flink Forward Global Virtual 2020
• 21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & CassandraCiesielczyk, Zontek · Flink Forward Europe 2019
• 40mStreaming ML with FlinkMarton Balassi · Flink Forward Berlin 2016
• 28mEnabling Machine Learning with Apache FlinkSherin Thomas · Flink Forward Europe 2019
• 37mBuild a Flink AI EcosystemJiangjie Qin · Flink Forward Europe 2019