Quick Guide on Data Quality Assurance
summary
Uses the Python library Great Expectations to code data quality checks end to end creating a Data Context YAML, wiring a datasource, authoring expectations, running validations and exporting results to a file that can be run in CI/CD. Measures six common dimensions including freshness completeness consistency accuracy validity uniqueness and sets concrete SLOs like 100% uniqueness or 90% completeness. Claims prior ML tuning showed 0% improvement until data quality was fixed and demonstrates an automation that produces repeatable validation reports.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 0:27Why data quality matters
- 3:15Key data quality dimensions
- 7:46Setting service level objectives
- 10:56Automating with Great Expectations
- 15:13Closing remarks
related talks
18mData quality: the key to long term happiness
41mData Quality Management: The Good, The Bad, and The Messy
50mNo more data quality surprises in your Databricks, Synapse or Power BI solutions!
44mData Quality Management
21mLarge-scale Data Quality Verification (...)
42mDeciphering the way data is tested
23mTests are docs and docs are tests: an introduction to great_expectations and data profiling
28mHands-on Workshop: How To Perform Data Quality At Scale