Build a Flink AI Ecosystem
summary
Claims a unified Flink-first ML stack that runs preprocessing, training and inference in one job by: a Table API-based ML pipeline (estimator/transformer API, aiming to add ~7 classic algorithms by 1.10), embedding TF and PyTorch as Flink operators (workers and PS become operators for fault‑tolerant distributed DL), and adding multi-variable and nested iterations plus mini-batch iteration to enable native SGD and continuous model updates for dynamic serving, with Python Table API and upcoming Python UDF support for user convenience.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker background
- 1:09Why build an AI ecosystem for Flink
- 8:16Lambda ML architecture with Flink
- 13:13Flink ML pipeline and FlinkML libs
- 19:35Integrating deep learning into Flink
- 24:23Enhanced iteration and dynamic model serving
- 31:48Python support in Flink
- 34:39Summary and ecosystem vision
- 35:19Q&A and closing remarks
related talks
38mAI made easy with Flink + AIFlow
20mUsing Flink for Versatile Feature Engineering Pipelines
40mStreaming ML with Flink
35mIntroducing Tensorflow on Flink
39mML Inference at scale, with Apache Beam and the Flink runner: Easy as learning your 5 times tables.
24mUnder Pressure: Applying ML in Real Time
21mIntroducing FlinkML
15mMetamodelization and Automatic Generation of ML Pipelines