Kedro + MLflow - Reproducible and Versioned data pipelines at scale
summary
A concrete demo shows a Kedro project bootstraps a pipeline with a declarative data catalog, node-based pipelines and an interactive visualization, then adds MLflow by adding an MLproject and Conda env and calling mlflow log_model and mlflow log_metric. The Iris example runs repeatedly, creating distinct run IDs and artifacts stored under mlruns, the MLflow UI surfaces metrics and models, and the same pattern deploys on Databricks for production.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 2:00Quantum Black mission
- 3:05Talk agenda
- 4:13What is Kedro
- 9:28Experiment tracking with MLflow
- 14:55Demo environment setup
- 17:27New Kedro project
- 19:22Kedro project structure
- 22:39Kedro data catalog
- 24:32Running Kedro pipeline
- 27:24Pipeline visualization
- 29:51Kedro Q&A
- 31:16MLflow introduction
- 33:04MLflow tracking server
- 35:38MLflow configuration
- 38:50MLflow integration
- 41:23Logging metrics and models
- 43:56How Kedro and MLflow work together
- 44:57Future of experiment tracking
- 45:40Resources and support
- 46:21Q&A
related talks
22mImprove Your ML Projects: Embrace Reproducibility and Production...
33mHow to build production-ready data science pipelines with
26mReproducible and maintainable data science code with Kedro
35mOrganizing data engineering around Kubeflow
31mMLflow: Platform for Complete Machine Learning Lifecycle
37mPlatform for Complete Machine Learning Lifecycle
17mIntroducing mlflow
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines