The LEGO Approach to designing PyData Workflows
summary
Claims an open‑source toolkit called DataJourney actually exists on GitHub, built over two years to stitch together interchangeable tools (ingest → monitor → visualize) using GitHub Actions, pre‑commit hooks and a background pipeline with pixie caching. Design method is concrete: an incremental “left‑shift” workflow using a 50/50 iteration ratio. The repo contains runnable commands and scripts; a live demo isn’t shown but the implementation is published and solicits issues.
AI-written summary · about
chapters (13)
- 0:00LEGO approach for PyData workflows
- 1:50Design thinking in data systems
- 3:18Building workflows with LEGO blocks
- 6:32Needs to business value in data
- 8:19Design-first vs tool-first in AI
- 10:06Left shift in data workflow design
- 12:37Incremental left shift for AI workflows
- 14:45Good data workflow design
- 17:32People first in data workflows
- 19:39Open source in workflows
- 22:34Data Journey framework
- 25:31Demo: Data Journey’s LEGO design
- 27:16Key takeaways
related talks
28mLego Bricks for Developers: Turning Insights Into Golden Paths
01h 14mUp your Bus Number - A Reproducible Data Science Workflow
24mMake data central feature
42mSimplifying Data Analysis with GitHub Codespaces, Jupyter Notebooks & Open AI
37mPatterns for Collaboration Between Data Scientists and Software Engineers
38mBuilding Data Pipelines in Python
33mBuilding Data Science Tools for Sustainable Transformation
33mAutomating Data Analytics: Collect, Transform and Deploy - Lead With Change