Holy D@t*! How to Deal with Imperfect, Unclean Datasets
summary
Specific, actionable tactics appear: use pandas backfill/ffill/fill for time series gaps and watch for hidden nulls like repeated nines; detect duplicate batches and create reliable unique keys; apply Dedupe and fuzzywuzzy for messy strings (example: matching "32 kilometers" to "32 km"); extract text with PDFMiner and OCR APIs and use speech recognition libraries for audio; apply functional constraints + a classifier and cost model to suggest repairs — a cited system retrains from user feedback and reaches ~90% accuracy.
AI-written summary · about
chapters (13)
- 0:00Messy datasets intro
- 1:26Know your data first
- 3:33Handling missing data
- 5:41Duplicated data entries
- 7:26Fixing inaccurate input
- 9:13Cleaning text strings
- 10:40Unicode encoding issues
- 12:09Standardizing measurements
- 13:16Outliers significance
- 14:37Unusual file formats
- 17:27Common sense in analysis
- 19:38Academic research on messy data
- 22:50Closing & resources
related talks
44mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
30mData-Centric AI Cookbook-let's prep that data
49mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
01h 31mPandas, Data Wrangling & Data Science
49mData Cleansing & Beyond: How to Address Data Debt for AI
32mEDA Techniques for Data Analysis
35mBad data - anecdotes and examples from the real world
01h 27mCleaning and Tidying Data in Pandas