KEYNOTE: You Are Overthinking Data Quality
summary
Argument: teams are overcomplicating data quality when specifics matter — what is the precise failure mode, what tests catch it, how many rows break it, and who is harmed. The talk favors property-based testing (aka data profiling) over fragile example-based oracles, recommends Great Expectations and fake data generators to scale tests, and describes setting measurable targets via service level objectives. It emphasizes removing intermediaries to learn actual user needs and target quality where it impacts people.
AI-written summary · about
chapters (11)
- 0:00Welcome
- 2:43Keynote introduction
- 5:30Oracle problem in testing
- 11:27Property-based testing fix
- 17:27Data profiling and testing
- 23:05Fake data costs
- 28:06Redefining data quality
- 34:07User impact understanding
- 41:06Balancing data factors
- 46:38Service level objectives
- 49:50Q&A
related talks
37mEngineering data quality
35mApplying Data Quality Monitoring
41mData Quality: The Holy Grail For A Data Fluent Organization
47mTest Data Key to Effective Test Coverage
52mScaling Data Teams: Three Pitfalls and How To Avoid Them
28mEngineering Data Reliably Using SLO Theory
16mHow to Prioritise Testing Activities to Reach Your Goals
41mA Cookbook for Testing Data