Giant Pile of Data
summary
Uses the recruiterspam dataset to show concrete clean-up and analysis steps: finds 1,500 unparsable messages, duplicate raw/parsed bodies and an unused experiments row to delete, notes the app runs on Heroku Postgres and breaks locally due to a custom Rails branch, recommends a one‑line script to restore backup and gemfile fixes, centralize logs and make DB read‑only for analysts, then runs a Mann‑Whitney U test on monthly counts and gets p≈0.93, rejecting the “first month of quarter has more spam” hypothesis.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:05Digital clutter and storage
- 4:34Keep only useful data
- 11:35Make data usable
- 16:29Use data to drive improvement
- 27:06Conclusion and contact information
related talks
46mData needs testing too
34mBuilding Data Driven Products with Ruby
45mData as culture as software
28mGenerate Insights, Not Excuses: How your data is slowing you down
19mZero Tolerance for Dirty Data: A Pointblank Prescription for Data Hygiene
44mMY AWFUL EX-DATASYSTEMS
02h 53mMining digital exhaust
24mHighly Scalable Discriminative Spam Filtering