Lessons from 6 months of using Luigi in production
summary
Luigi runs dependency DAGs for ~500 jobs feeding a warehouse that supports 2000+ reports and 400+ BI users; Cron was replaced by EC2-based Luigi workers, with successes and surprises: fragile pipelines that nuke-and-reload the whole schema, a shift to per-table schema hashes to avoid global drops, adding unit/test modes via Luigi parameters, encapsulating larger DB transactions to avoid partial failures, and emitting data-quality metrics to monitoring/Slack so operational problems become actionable.
AI-written summary · about
chapters (27)
- 0:00Introduction and speaker background
- 2:29Why Luigi over pager duty alarms
- 2:49Building the initial data pipeline with Luigi
- 8:00Why Luigi solved the dependency problem
- 10:29Automating the Luigi pipeline in production
- 11:12Handling frequent schema changes in production
- 12:35Nuking the database and starting fresh
- 12:57Adding test and unit modes to Luigi
- 14:01Importance of fast and competitive testing
- 14:44Integrating external data sources like Staffomatic
- 16:27Designing clean and testable data loading classes
- 17:30From data science to business critical reports
- 18:33Why reports fail and how to fix them
- 20:20Isolating critical tasks in the DAG
- 21:01Handling failures in financial data pipelines
- 22:07Using database transactions for atomic tasks
- 22:48Monitoring data quality and outcomes
- 24:30Scaling from UK to global operations
- 25:34Lessons learned and future improvements
- 26:58Final thoughts and takeaways from six months
- 29:26Closing remarks and call for data engineers
- 30:11Q&A session part one
- 34:39Q&A session part two
- 35:37Q&A session part three
- 37:38Q&A session part four
- 39:03Q&A session part five
- 40:26Q&A session part six
related talks
32mLarge-scale machine learning pipelines using Luigi,...n
24mLuigi & Data Pipelines
44mCreating Python Data Pipelines in the Cloud
16mReplacing Makefiles with Python and Luigi
32mBuilding Your First Data Pipelines
38mData Secrets from a Platform Engineer
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
14mHere, There and Everywhere: Spotting Data Opportunities 101