What are data unit tests and why we need them
summary
Defines data unit tests as explicit expectations on production datasets (not code), implemented with libraries like Great Expectations (also compares Pandera TensorFlow Data Validation Soda), demonstrated by profiling a red‑wine set then validating a white‑wine set where a distribution divergence check fails. At one company these tests run in CI daily health checks across ~15 data sources, catch duplicates missing columns or wrong fields, sometimes add hours to deployment but prevent bad releases.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:35Agenda
- 1:20Theodore Meynard
- 2:01Data unit tests
- 3:52What are data unit tests
- 5:27Data unit test frameworks
- 17:11Incidents
- 18:21Inhouse validator
- 19:16Conclusion
- 24:33Documentation
- 26:49Use cases
