Data needs testing too
summary
A practical playbook for treating data like code: load messy files into Postgres with Postgres COPY, profile a 26,000-row contact file to spot a unique club‑card key and email/phone failures, deduplicate with Levenshtein edit distance, and validate phones with a lightweight libphone-style check that finds 2% missing 3% errors. Add DB constraints, split JSON/XML into fields, and run automated nightly SQL tests (about 100/day) to catch real incidents — e.g. a detected spike from the Grand National rather than a system bug.
AI-written summary · about
chapters (15)
- 0:00Data testing importance
- 1:03Data sources in apps
- 3:28Spreadsheet data corruption
- 5:16API and device data reliability
- 9:48Defect distribution in components
- 12:13Testing files and databases
- 19:54Validating email addresses and phone numbers
- 22:01Handling duplicate data
- 24:33Missing and misplaced data
- 26:41Monitoring data anomalies
- 31:43Data testing vs performance
- 33:26Improving data quality
- 37:27Trapping data errors
- 39:11Data validation tools
- 40:26Q&A
