Beyond scraping: how to use machine learning when you're not sure where to start
summary
Builds a system that extracts the publication date of almost any news article by framing it as supervised classification of DOM elements, using crowdsourced labels (three annotators, drop disagreements), feature engineering from heuristics (contains “published”, author proximity, meta and class attributes), and practical production choices (NLTK + LIBSVM, avoid headless rendering for speed). Results land around ~90% accuracy across ~550 publications, with a big boost from a simple URL-date heuristic.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:25Beyond scraping
- 1:28Extracting date of a second page: also easy
- 2:19Could get people to do it
- 3:10Get a programmer to do it
- 4:14The big guns: get machine learning to do it
- 5:00But: machine learning is a PITA
- 6:59How do humans recognize publication date?
- 11:59The hardest part of machine learning...
- 12:32Enter this research paper
- 12:57supervised learning/ classification
- 13:55Training a model
- 14:50Getting data
- 17:29get all DOM elements w dates
- 18:01Feature selection
- 18:27Training the model
- 19:01tutorial
- 20:05extracting date from DOM element as a string
- 21:56Differences between prod and research Major ne performance issues in production Did not implement headless browser features, too slow for production paring
- 22:21Lessons learned
- 23:34Machine learning is about thinking statistically...
related talks
31mHow Soon is Now: extracting publication dates with machine learning
34mApplying Machine Learning Techniques to Detect and Understand Fake News
01h 17mOne workshop that data scientists don't want you to attend...
36mMachine Learning in News Media: Case study 24sata.hr- 34mApplying Machine Learning Techniques to Detect and Understand Fake News
35mUnderstanding the News around the World with Web Scraping and NLP at Scale
48mHow to make intelligent web-apps
26mUnsupervised learning for news summarisation