Production-ready Data Science with Python and Luigi
summary
Concrete production pipelines built with Spotify Luigi are shown, using runnable Python Tasks implementing parametrize and output to wire download → train → evaluate → export steps. Examples include image-processing and training pipelines, containerized runs, Spark integration, CI/CD hooks, a baseline validation step that gates deployment, and a Luigi server visualizer; models around 4.5 GB are evaluated and exported end-to-end in the demo.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 10:05Jupyter in production
- 24:29Luigi pipelines
- 37:11Workflow parameters
- 50:36Model validation & security
- 1:06:10Deployment strategies
- 1:14:56Q&A
related talks
24mLuigi & Data Pipelines
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
31mHow to quickly build Data Pipelines for Data Scientists
42mBuilding a Data Driven Organization
01h 03mBig Data Pipelines With Scala - Part 1 of 2
40mPreparing data for LLM training with Data Prep Kit
45mWorkshop 3: Data Engineering - A Roadmap... by Ramon Perez [Part 3]