Thinking of Topic Modeling as Search
summary
Claim: convert topic models into search queries by turning documents and topics into dense vectors, store and query them in Elasticsearch or Redis, and retrieve current posts without tags. Pipeline specifics: use sentence-transformers to make embeddings, prefer localized keyword-based topic embeddings over simple means, measure with precision@K and recall@K, cache topic vectors, and run dual time windows (≈180 days evergreen vs 7-day emergent). Preprocessing (strip PII, filter replies, handle numbers) proves essential and the keyword-localized change noticeably fixes noisy retrieval.
AI-written summary · about
chapters (14)
- 0:00Topic modeling project intro
- 1:30Real-time topic modeling
- 2:52Search as topic modeling
- 4:37Text embeddings in topic models
- 7:07Building the pipeline
- 8:32Training from scratch
- 9:36Evaluating model performance
- 11:40Embedding failures
- 13:46Embedding design lessons
- 14:07Pre-processing pitfalls
- 15:32Refining keywords with TF-IDF
- 16:55Tracking topics over time
- 17:56Conclusion and applications
- 18:57Q&A
related talks
13mThinking of Topic Modeling as Search
49mTopic Modeling: A Provable Spectral Method
26mImplementing a Search Index
41mResearch in the Age of AI
26mAdvancing Multi-Modal Search Capabilities in Search Pipeline
30mSearch on Steroids — A Story of Semantics
25mSemantic Search
30mSemantic Search and Neural Information Retrieval