How to not pull your hair out while providing data to the business
summary
Claims that applying unit testing for data pipelines in Python prevents breaking dashboards and models by testing a refactored transform function end to end. Uses Pandas, Pytest and CI (GitHub Actions/GitLab CI/Azure) with synthetic input/output data, a five‑step refactor → test process, and a beer dataset example. A naive split produced two rows for a non‑alcoholic beer; changing to rsplit plus strip fixes it and tests turn green before deployment.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:01Why automated testing?
- 5:11Setup
- 8:16Refactor
- 12:37Create a test
- 17:12Then change the implementation
related talks
25mTesting Data Pipelines
39mUnit Testing for Data Scientists
01h 30mAdvanced Software Testing for Data Scientists
01h 29mAdv. Software Testing for Data Scientists
39mModern Data Pipelines Testing Techniques: A Visual Guide
38mBuilding Data Pipelines in Python
39mTesting Big Data Applications (Spark, Dask, and Ray)
20mShift-left testing : Building reliable Data Pipelines