Zero Tolerance for Dirty Data: A Pointblank Prescription for Data Hygiene
summary
Shows a concrete Pointblank workflow: use scan_data to get an HTML overview, create an agent interrogation with col_vals expectations to halt on failures, and use get_data_extracts to pull failing rows. Finds 400 records with negative points (entered as -3), 163 invalid play types and ~1,000 invalid opponents, fixes the 2008 dataset (lakers_2008_fixed) to get all passing steps, then uses yaml_write / yaml_read_agent to port the validation to 2009 and produce correct top-ten scores.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:30Loading data with pointblank
- 3:12Scanning data quality
- 4:57Filtering vs expectations
- 6:42Agents for validation
- 8:31Investigating data issues
- 10:17Extracting valid data
- 11:26Comparing validation methods
- 12:29Documenting in YAML
- 13:52Porting validation recipes
- 17:47Summary
related talks
46mData needs testing too
18mData quality: the key to long term happiness
29mGiant Pile of Data
41mA Cookbook for Testing Data
15mUsing Statistical Techniques to Automatically Detect Game-Breaking Issues
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
45mDo those numbers look right? How to ensure quality, and avoid inaccuracies in your data insights.
39mDoing Data: The Critical Process of Data Preparation (...)