Large-scale machine learning pipelines using Luigi,...n
summary
Claims a runnable, production-grade pipeline: Luigi + PySpark + scikit-learn orchestrates ETL, feature extraction, training and evaluation for demand forecasting across ~30 countries, 11,000 stores and thousands of articles, generating model per article per country artifacts and storing configs for reproducibility. Experiment turnaround drops from multi-day runs to incremental re-runs, but pain points remain: HDFS file-existence latency, Spark startup overhead, dangerous manual output deletes. Works for scheduled batch training and scoring — batch training not streaming.
AI-written summary · about
chapters (11)
- 0:00Start
- 2:08Agenda
- 2:52Machine Lewning Pipelines
- 4:57Scaling up
- 6:42Team Experience
- 8:37Are we reinventing the wheel?
- 9:29Purpose of Luigi
- 10:24Philosophy
- 11:02Everything is Python
- 14:47Dynamic task generation
- 15:28Pipeline example
related talks
35mMachine Learning Pipeline using Luigi and Scikit Learn
42mLessons from 6 months of using Luigi in production
23mRethinking scalable machine learning with Spark ecosystem
32mBuilding Your First Data Pipelines
24mLuigi & Data Pipelines
39mMachine Learning with Scala on Spark
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
44mCreating Python Data Pipelines in the Cloud