How to do real TDD in data science? A journey from pandas to polars with pelage!
summary
Claims that Pelage makes real TDD feasible in data workflows and that its Polars integration stays lightweight. The demo builds an open‑source testing layer on top of Polars and shows concrete steps: refactor for method chaining, translate from pandas to Polars, write failing checks, then add minimal fixes until tests pass. Surprising findings surface (masses >2000 kg, ~1M rows with three nulls, duplicated primary keys) and a ~4× speed gain appears; the live coding and package work end‑to‑end.
AI-written summary · about
chapters (19)
- 0:00Real TDD in data science
- 1:06What is TDD and why data science
- 2:55Refactoring pandas with method chaining
- 10:21Pandas to polars efficiently
- 13:53Introducing Page for TDD
- 14:58First failing tests with Page
- 16:06Enforcing data types with Page
- 17:12Removing nulls in categoricals
- 17:56Testing commercial names
- 19:01Validating primary keys
- 20:03Standardizing fuel types
- 21:30Filtering invalid fuel types
- 22:35Summary of Page features
- 23:21Method chaining in data science
- 24:23Deploying Page in pipelines
- 25:24Q&A: Page with pytest
- 27:12Q&A: Rust integration for Page
- 28:37Q&A: Page vs dbt timing
- 30:49Q&A: Page vs Great Expectations
related talks
58mFrom Pandas to Polars Upgrading Your Data Workflow
39mWhat we learned by converting a large codebase from Pandas to Polars
01h 32mAn Introduction to Polars
30mIs the great dataframe showdown finally over? Enter: polars
28mExpressive & fast dataframes in Python with polars
38mExpressive and fast dataframes in Python with polars
01h 11mTest-Driven Data Analysis
29mTDD shouldn't be TDDious