Dataframely — A declarative, 🐻❄️-native data frame validation library
summary
Dataframely is a Polars-native validation library that ships declarative schemas, column checks and cross-column constraints expressed as Polars expressions, and returns either errors or partitioned rows via hard validation / soft filtering. It supports primary keys, group-by rules across related frames in a collection, automatic casting, typed DataFrame generics for static type hints, sample data generation for tests, and has run in QuantCo production for over a year improving legibility and robustness while remaining native-fast. Versioning and failure utilities give actionable counts and failing-row extraction.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:25Why we built Dataframely
- 3:31Data pipeline validation challenges
- 5:39Hospital claims example
- 6:42Defining schemas
- 8:27Cross-column constraints
- 9:33Validate and filter
- 13:06Typed data frames
- 15:11Interdependent data frames
- 18:45Testing pipelines
- 20:31Pipeline impact
- 22:26Q&A
related talks
30mIs the great dataframe showdown finally over? Enter: polars
32mPandas, Polars and the DataFrame Consortium.
28mExpressive & fast dataframes in Python with polars
51mThe middle way for static typing in Spark DataFrames
22mValidating and Testing R Dataframes with Pandera via reticulate
38mExpressive and fast dataframes in Python with polars
06mInterview
14mMaking Data Quality an integral part of developing Machine Learning and Data Products