Automated Scientific Pipelines With Doit

on YouTube ↗25m

" summary

doit is a Python task manager that claims to replace Make with pure Python task definitions and can run as a CLI or via its API. examples show a dodo.py that downloads 3 data files (latest months), generates per-file plotting tasks, and stores hashes in an up-to-date database to skip work. tasks are yielded programmatically, mix shell and function actions, run in order or parallel, and the demo completes and produces plots.

AI-written summary · about

" chapters (6)

1. 0:00Intro
2. 0:25What is a pipeline
3. 4:12Doit
4. 8:08Command Actions
5. 15:22Running Tasks
6. 21:19Questions

" related talks

• 38mBuilding Data Pipelines in PythonMarco Bonzanini · PyData London 2016
• 38mBuilding Data Pipelines in PythonMarco Bonzanini · PyData London 2016
• 29mDagster: a Framework for Data Processing ApplicationsNick Shrock · Scale By The Bay 2019
• 26mCLI, CI, DIY: Crafting Python Based Dev Tools in a DevOps VacuumLiubov Burtseva, Uri Drori · PyCon Israel 2024
• 11m"Projit: An Open Source Python Tool for Decoupled Data Science"John Hawkins · PyCon AU 2024
• 01h 05mPython MapReduce Programming with PydoopSimone Leo · EuroPython 2011
• 47mDagster: An open source Python library for building data applicationsNicholas Schrock · Crunch Conference 2019
• 44mCreating Python Data Pipelines in the CloudFemi Anthony · PyData DC 2016