Unsupervised learning for news summarisation
summary
Unsupervised pipeline extracts topics from ~400 articles per day, built for Polish and running in production at the Chancellery for eight months. It scrapes with Scrapy, preprocesses with tidytext, scores bursty terms with a log‑likelihood test (keep stats >10 and top 10% frequent words), forms SVD embeddings, clusters by agglomerative hierarchy with silhouette to pick topic count, and ranks sentences using LexRank with log‑likelihood instead of IDF, de‑duplicates and outputs compact summaries without GPUs.
AI-written summary · about
chapters (16)
- 0:00Unsupervised news summarisation
- 1:11Unsupervised learning for Polish news
- 2:38Key terms in summarisation
- 3:46Model pipeline overview
- 5:33Important words with log-likelihood
- 8:04Word embeddings via SVD
- 9:31Topic clustering with hierarchy
- 13:13LexRank for multi-document
- 16:52Sentence selection and deduplication
- 18:00Model on GitHub
- 18:23Q&A: Python and R
- 19:36Q&A: Model deployment
- 21:05Q&A: TextRank comparison
- 22:23Q&A: Missed events oversight
- 23:11Q&A: Word embeddings source
- 24:39Q&A: Scalability and integration
related talks
35mUnderstanding the News around the World with Web Scraping and NLP at Scale
21m"What's in the news?" #bbuzz
58mAutomatic text summarization
22mEmpowering journalism-approaches to automatic text summarization
39mUnsupervised NLP Classification with Clustering
30mProductionizing an unsupervised machine learning model to...
43mA Case Study of Natural Language Processing in Clojure
38mAutomatically categorizing news articles on nrk.no