Democratization of ML Pipelines and Bringing ML Workflows to Heterogeneous...
summary
Kubeflow Pipelines v2 introduces an Intermediate Representation (IR) that is platform‑agnostic and already used by Kubeflow Pipelines, Google Vertex AI and Tekton, plus a backend compiler and abstract plugin layer to target Argo, Tekton or other runtimes. Caching and metadata move into a centralized ML metadata service with driver/publisher executors, cutting pod creation and reducing a demo pipeline from ~30 seconds uncached to ~2 seconds fully cached. The system is implemented and shown working in the UI and SDK.
AI-written summary · about
chapters (11)
- 0:00Democratizing ML pipelines
- 1:05ML workflow automation challenges
- 3:29Heterogeneous pipelines on Kubernetes
- 8:48Kubeflow Pipelines V2 design
- 11:59Pipeline intermediate representation
- 16:40Decoupling from workflow engines
- 19:29ML metadata improvements
- 23:45Performance optimizations in V2
- 28:44Demo of Kubeflow Pipelines V2
- 33:31Future roadmap for Kubeflow Pipelines
- 35:19Resources and community
related talks
31mBringing ML Workflows to Heterogeneous Cloud Native Machine Learning Pla...
29mML Engineering Kubeflow
09mKubeflow Pipelines v2: The Next Generation of MLOps on Ku...
54mScaling Machine Learning Pipelines in Cloud
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
35mOrganizing data engineering around Kubeflow
14mKubeflow: Where ML Meets Kubernetes
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow