Introducing Tensorflow on Flink

on YouTube ↗35m

" summary

An implementation that runs TensorFlow (and PyTorch) inside Flink by exposing machine‑learning worker/PS roles as Flink operators and an application manager operator that tracks heartbeats and restarts nodes, so training and inference run without writing intermediate data out. Java↔Python IPC uses a shared‑memory memory queue with about 10% overhead. Supports batch and stream modes, dynamic parallelism (e.g., set operator parallelism to spawn N workers), failure recovery via Flink restarts plus AM‑driven node restarts, and integration work with PyFlink and Flink operator coordinator.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 1:15Background and motivation
3. 6:24TensorFlow on Flink solution
4. 10:39Running TensorFlow on Flink
5. 21:28Generalizing to Flink AI Extended
6. 24:39ML framework and operator
7. 30:49Summary and future work

" related talks

• 37mBuild a Flink AI EcosystemJiangjie Qin · Flink Forward Europe 2019
• 41mPredictive Maintenance with Apache FlinkDongwon Kim · Flink Forward Berlin 2017
• 34mWorking with Tensorflow from the JVM #bbuzzHolden Karau · Berlin Buzzwords 2018
• 21mIntroducing FlinkMLBoris Lublinsky, Stavros Kontopoulos · Flink Forward Berlin 2017
• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 38mAI made easy with Flink + AIFlowJiangie (Becket) Qin · Flink Forward Virtual 2020
• 21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & CassandraCiesielczyk, Zontek · Flink Forward Europe 2019
• 18mMachine learning with Flink in WeiboQian Yu · Flink Forward Virtual 2020