Machine Learning Pipeline using Luigi and Scikit Learn
summary
Builds a Luigi-orchestrated ML pipeline that uses Scikit-Learn to train a model, persists intermediate CSV states and enforces atomic file operations, and produces a pickled model that a small Flask app serves via a Flask predict endpoint with a reload API to hot-reload updated models. Demo runs on a Kaggle-style retail dataset, shows Luigi’s dependency graph, retries, and central scheduler UI. Notes: Luigi lacks a built-in scheduler (cron used) and Scikit-Learn parallel jobs can fail when n_jobs>1 under Luigi’s process assignment.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:15Data Preparation
- 2:29Python Code
- 8:42Directed Acyclic Graph
- 11:37Luigi
- 13:01Luigi Task
- 14:49Luigi Data Preparation
- 15:53Data Aggregation
- 17:38Train
- 18:45Execution
- 20:40Demo
- 26:42Predict
related talks
32mLarge-scale machine learning pipelines using Luigi,...n
43mDeploying Machine Learning using sklearn pipelines
28mDynamic Data Pipelining with Luigi
43mMachine Learning Engineering principles with Python and MLFlow
16mReplacing Makefiles with Python and Luigi
54mScaling Machine Learning Pipelines in Cloud
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
37mSoftware Development for Machine Learning in Python