A Deep Dive into Kubeflow Pipelines
Talkat Kubernetes AI Day Europe 2022 · on Monday, 2022-05-16· at 15:25· Pavilion 3, Room D | Level 2 | Central Forum·
summary
Overview of Kubeflow Pipelines architecture with concrete details: the pipeline service parses Python DSL and compiles YAML, metadata sits in a MySQL metadata store, artifacts default to MinIO artifact repo but can use S3/GCS, each pipeline step runs as a Kubernetes pod, Argo is the orchestration controller and uses the Emissary executor (preferred over Docker), SDK v2 adds a pipeline root and step caching, multi-user isolation maps profiles to namespaces, demo shows a retrain loop with one pod per step.
AI-written summary · about
chapters (11)
- 0:00Start
- 3:33Agenda
- 4:42ML Workflow
- 6:21ML Pipeline Tools
- 8:08Kubeflow - The Machine Learning toolkit for Kubernetes
- 10:30Kubeflow - Installation Options
- 11:24Kubeflow Pipeline Platform Components
- 13:02Kubeflow Pipeline Architecture
- 18:41Choosing an Argo Workflows Executor
- 20:28Other notable features of KFP
- 23:26Kubeflow pipeline demo
related talks
29mML Engineering Kubeflow
36mMachine learning pipeline Kubeflow, let's start
35mOrganizing data engineering around Kubeflow
54mScaling Machine Learning Pipelines in Cloud
09mKubeflow Pipelines v2: The Next Generation of MLOps on Ku...
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
31mKubeflow - Data Science on Steroids