Building an End-to-End Analytics Pipeline with PyFlink
summary
A runnable pipeline that reads mailing‑list rows via the Postgres CDC connector, classifies each message with an LDA model trained locally using gensim and spaCy exposed as a PyFlink Python UDF, and writes results back to Postgres via the JDBC sink is demonstrated; the job runs as a continuous Flink stream, models require tuning (one topic dominates oddly) and dashboards are shown with superset visualization. PyFlink is installable from PyPI and supports deploying model files to task managers.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:40Why Python
- 8:20Change Data Capture
- 12:52Create Table
- 14:22Create UDF
- 15:59Create Query
- 18:45Submit Job
- 20:15Visualization
- 21:42Improvements
- 23:03Summary
related talks
25mBuilding a Python Data Pipeline with Apache Flink
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
37mBuild a Flink AI Ecosystem
01h 21mBuilding a ML Model Observability pipeline
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
34mBuilding End-to-End Machine Learning Workflows with Kubernetes...
35mPyFlink: Building Flink applications with Python language
40mIt’s finally here: Python on Flink & Flink on Zeppelin