How Soon is Now: extracting publication dates with machine learning
summary
A parser implements a research paper with supervised learning to extract publication dates across sites, training on human-labeled HTML snippets and features like class names, proximity to title, and pixel heuristics; it uses lxml DOM chunking, regex to pull date tokens, and an NLTK NaiveBayes classifier (scikit‑learn alternatives noted). Production drops slow headless-render features, prefers a URL date override when present, and resolves ties by taking the first positive chunk. Deployed at scale (≈1M pages/week, 550k publications) it attains about 88% precision versus the paper’s ~96%.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:05Date extraction challenges
- 3:13Machine learning for dates
- 4:37Human cues for dates
- 8:05Setting up ML
- 10:55Gathering training data
- 13:58Splitting pages
- 15:01Feature selection
- 17:07Training classifier
- 18:50Selecting correct date
- 20:16Extracting final date
- 21:40Post-filtering edge cases
- 23:26Results and accuracy
- 24:33Production vs research
- 25:57Lessons learned
- 29:25ML vs traditional algorithms
- 30:29Q&A
related talks
25mBeyond scraping: how to use machine learning when you're not sure where to start
30mPerforming Content: Can NLP and Deep Learning algorithms predict reader prefere...
34mApplying Machine Learning Techniques to Detect and Understand Fake News
35mUnderstanding the News around the World with Web Scraping and NLP at Scale
22mPython for science, side projects and stuff!
49mDiscovering Research Ideas Using Semantic Vectors & Machine Learning- 34mApplying Machine Learning Techniques to Detect and Understand Fake News
26mUnsupervised learning for news summarisation