pydiverse pipedag: A library for data pipeline orchestration
summary
A Python library that wires real pandas polars sqlalchemy and ibis code into a DAG and claims automatic cache invalidation so only changed steps rerun, with support for lazy SQL execution to push work into the database, hooks for polars and raw SQL, and a flow/run separation that enables many short local instances and CI usage. It is already used at quanco and emphasizes many pipeline instances and gradual refactoring rather than big‑bang rewrites.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 4:21Data pipeline orchestration
- 6:35Hello world with pipedag
- 11:15Typical pipeline structure
- 16:17High iteration speed
- 19:51Developing with many instances
- 24:22Fast iteration principles
- 26:09Future of pipedag
- 26:35Q&A
related talks
36mPlumbing in Python: Pipelines for Data Science Applications
28mYour Pipeline as a Function.
23mWhat would the django of data pipelines look like? with Lisa Dusseault
32mSimple Data Engineering in python 3.5+ with Bonobo
38mFrictionless Data Pipelines
36mElegant data pipelining with Apache Airflow
15mVisualizing Data Analysis Pipelines with Pandas Tutor and Tidy Data Tutor
27mLet Your Data Tell Its Story