DDataflow: An open-source end to end testing from machine learning pipelines
summary
DDataflow is an open-source tool that enables end-to-end tests by applying decentralized sampling per data source so CI runs the real pipeline against a small realistic dataset. Implementation targets PySpark and is currently Spark-only, with modes for CI, production, and local download. It uses a filtered subset of production data (not synthetic) and ships sampling configs per source; deployed at Geto guide it shortens feedback loops and makes full-pipeline CI practical.
AI-written summary · about
chapters (9)
- 0:00DDataflow introduction
- 1:23Testing ML pipeline challenges
- 3:55Unit testing limits for ML
- 7:50End-to-end tests to DDataflow
- 9:19Data sampling strategies
- 11:49DDataflow pipeline integration
- 14:00DDataflow production impact
- 15:50DDataflow live demo
- 19:48Q&A
related talks
25mTesting Data Pipelines
39mTesting Big Data Applications (Spark, Dask, and Ray)
05mData Quality and Validation in ML Pipelines
29mDagster: a Framework for Data Processing Applications
31mHow to quickly build Data Pipelines for Data Scientists
17mProductionizing Data Science Workloads
46mBuilding a Self-Managing Test Data Marketplace
34mTesting Big Data Applications (Spark, Dask, and Ray)