Understanding the News around the World with Web Scraping and NLP at Scale
summary
Pipeline scrapes HTML, saves pages to S3, runs Python NLP (spaCy NER plus custom models), links mentions to Wikidata using kensho-processed Wikipedia CSVs, and ranks importance with a position‑rank / salience algorithm; results land in Snowflake via Snowpipe so metadata in ~3 seconds is typical. Translation uses LibreTranslate with GPU speedup from about 30s → 5s in tests. The team favors heuristics and open source over slow embedding lookups like vector search, e.g. kensho Wikipedia CSVs.
AI-written summary · about
chapters (15)
- 0:00News analytics with Tarpie
- 1:08News readership tracking
- 2:53Web scraping pipeline
- 5:42Entity extraction with SpaCy
- 7:06Entity linking with graphs
- 10:13Wikipedia for disambiguation
- 13:48Entity resolution pitfalls
- 15:55Key entities in articles
- 18:03Key terms with TextRazor
- 19:28News categorization with IPTC
- 21:57Global news translation
- 24:28News analytics pipeline
- 25:32Year-end readership trends
- 25:53Processing unstructured data
- 27:19Q&A
related talks
26mUnsupervised learning for news summarisation
44mTeaching Computers ABCs: A Quick Intro to Natural Language Processing
30mScaling Embeddings in Real-Time
34mApplying Machine Learning Techniques to Detect and Understand Fake News
21m"What's in the news?" #bbuzz
23mMultilingual Natural Language Processing
24mText Analytics Simplified
31mNLP text recommender system journey to automated training pipeline with Spark and Sagemaker