Tests are docs and docs are tests: an introduction to great_expectations and data profiling
summary
Introduction to using Great Expectations as a Python library to write declarative “expectations” (e.g. column week_of_month between 1 and 5, room_temperature 60–75°F) that serialize to JSON, render as HTML docs, and run against pandas or Spark batches in CI/CD. Demonstrated a profiler that uses ML (linear regression) and heuristics to detect deterministic relations (age ↔ year_of_birth), surface rare-label typos, and auto-generate expectations with a 90% acceptance ratio threshold; profiler is work‑in‑progress but produces usable tests and docs.
AI-written summary · about
chapters (7)
- 0:00Introduction to talk
- 1:10What is data testing
- 4:00Great Expectations overview
- 6:48Creating expectations manually
- 10:21Automating data documentation
- 13:34Data profiling concept
- 19:15Q&A and closing
related talks
15mQuick Guide on Data Quality Assurance
42mDo Your Homework! Writing tests for Data Science and Stochastic Code
50mNo more data quality surprises in your Databricks, Synapse or Power BI solutions!
32mWrite Less and Test More with Data Regression Testing
01h 29mAdv. Software Testing for Data Scientists
29mTDD shouldn't be TDDious
34mWhat are data unit tests and why we need them
35mTesting: Choose Your Weapons