How to build production-ready data science pipelines with
summary
Kedro structures code with a project template, YAML data catalog and node/pipeline functions, and integrates with MLflow for model artifacts and versioning. A production deployment uses a Python package on a VM hooked to a SQL database via credentials in configuration, or alternatively Docker images or copied source on GitHub. A real-world build is an end-to-end Kedro optimization pipeline for a Dutch horticulture client that runs in ~30 minutes after ~3 months development and delivers 10-15% energy cost reduction.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:12Why Kedro
- 5:00Kedro overview
- 6:54Project template
- 9:03Data set support
- 14:45Who is FAA
- 16:51Pipeline after development
- 19:21Deployment options
- 23:00Delivery results
- 24:02Does the client see the added value
- 29:05Deployment
related talks
22mImprove Your ML Projects: Embrace Reproducibility and Production...
49mKedro + MLflow - Reproducible and Versioned data pipelines at scale
26mReproducible and maintainable data science code with Kedro
40mData Science Best Practices with Kedro
35mOrganizing data engineering around Kubeflow
32mMLOps at Renault: A Generic Pipeline for Scalable Deployment
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
41mDeploying machine learning pipelines to production