Lifecycle of Web Text Mining: Scrape to Sense
summary
Scrapes Pillreports.net with urllib2 and BeautifulSoup and stores cleaned records in MongoDB via pymongo, keeping reports and comments separate; language detection filters English entries. Shows a downward trend in warnings, uses bag‑of‑words + nltk preprocessing, chi‑square to pick top features and visualizes word counts versus warning labels; a simple Naive Bayes yields ~73% training and ~72% test accuracy on descriptions and ~90% when using consumed user reports; PCA first two components explain ~3.5% and reveal cluster and content signals like sour taste → PMMA warning and feature selection by chi‑square.
AI-written summary · about
chapters (14)
- 0:00Web text mining intro
- 1:16Why pillreports.net data matters
- 4:29Scraping and cleaning dataset
- 7:22Report origins and languages
- 9:13Python visualization challenges
- 10:38Trends in warning reports
- 11:44Text classification model
- 13:33Chi-square feature selection
- 15:24Visualizing warning-related words
- 17:28Pill composition effects insights
- 18:32User reports after consumption
- 20:17PCA and language patterns
- 22:03Word clouds for drug experiences
- 23:34Q&A
related talks
01h 51mNatural Language Processing in Python
35mUnderstanding the News around the World with Web Scraping and NLP at Scale
32mWeb Scraping Best Practises
25mBeyond scraping: how to use machine learning when you're not sure where to start
29mUn Enfoque Pythonico para Minería de Textos de Historias de Trabajo Arduo, Migración y Resiliencia
25mFrom MCP to Scale: Pipelines That Build Themselves
45mThreat Hunting with Python and Pandas
45mData Science: It's Easy as Pyǃ