Bad data - anecdotes and examples from the real world
summary
Catalog of real failures and fixes: concrete examples of corrupted CSVs, mixed units, locale date swaps that turn March into May, and typos that split categories; proposed causes like repurposed legacy data and manual Excel workflows; practical fixes including sqlite for local staging, datasette for quick exploration of transformed tables, and explicit assertions in notebooks to fail early; notes on PDF/OCR pain and recommendation to capture provenance and validation at ingestion so downstream metrics actually match reality.
AI-written summary · about
chapters (17)
- 0:00Bad data introduction
- 1:12Why bad data matters
- 2:14Origins of bad data
- 4:44Good data principles
- 5:49Bad data sources
- 9:04Directory naming issues
- 12:11CSV missing headers
- 14:16Categorical value issues
- 16:46Unit mismatches
- 17:49Unvalidated data sources
- 18:54Multinational data entry
- 20:02Date format errors
- 21:31Handling messy data
- 24:20Data tools
- 27:58Efficient transformations
- 29:04Key takeaways
- 30:54Q&A
related talks
44mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
49mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
30mData-Centric AI Cookbook-let's prep that data
29mHigh Performance Data Loss
28mWhat Data Analysts Wish Application Developers Knew
01h 29mImproving Open Data Quality using Python
01h 31mPandas, Data Wrangling & Data Science
28mThe bumps in the road: A retrospective on my data visualisation mistakes