What would the django of data pipelines look like? with Lisa Dusseault
summary
Proposal: a small framework called Phaser that applies Django ideas to pipelines and that is already used to prototype a messy HR ingest. It provides declarative column and step declarations, lets custom logic be a row-by-row decorated step, and runs pipelines as phases with checkpoints and CSV diffing so you can inspect row numbers and intermediate files. Built-ins include uniqueness checks, drops, casting and renaming for CSV and nested JSON, run-tools for local runs, and guidance to drop bad rows early; prototype works but not widely deployed.
AI-written summary · about
chapters (9)
- 0:00Web frameworks to pipelines
- 3:56Introducing Phaser
- 7:29Declarative steps and columns
- 9:59Pipeline phases and flow
- 12:43Run tools and exceptions
- 14:27Utilities and consistency
- 16:34Extensibility and Django
- 18:01Hope for data tools
- 20:30Q&A
related talks
34mpydiverse pipedag: A library for data pipeline orchestration
36mPlumbing in Python: Pipelines for Data Science Applications
38mFrictionless Data Pipelines
20mBuilding a Data Pipeline with Testing in Mind
27mA Data Harmonization Engine on top of Django - The Good, the Bad, and the Ugly
42mBuilding a Data Driven Organization
31mHow to quickly build Data Pipelines for Data Scientists
28mData Plumbing 101 - ETL Pipelines for Everyday Projects