Thinking of Topic Modeling as Search
summary
Treat topic modeling as search by training BERTopic daily, saving document embeddings in a vector store (e.g. PGVector or OpenSearch), and creating topic centroid embeddings to query that store. Using the mean of all topic docs returns noisy, near‑random hits; localizing the centroid with class‑based TF‑IDF keywords, nearest‑doc cosine selection, or an LLM summary yields focused results. Prototype returns about one in four documents preassigned to a topic and top results are reliably on‑topic; image captioning fills textless posts.
AI-written summary · about
chapters (8)
- 0:00Introduction to topic modeling
- 1:07Why search instead of tagging
- 3:35How embeddings work
- 5:24Storing embeddings in databases
- 6:49Training the topic model
- 8:16Creating topic embeddings
- 9:44Testing the search approach
- 12:52Key takeaways and conclusion
related talks
29mThinking of Topic Modeling as Search
45mDecoding Topics: A Comparative Analysis of Python’s Leading Topic Modeling Libraries Using Climate C
49mTopic Modeling: A Provable Spectral Method
27mUnveiling Clustering in BERTopic Topic Modeling
34mTopic modeling with BERTopic
47mThe Neural Search Frontier #bbuzz
26mImplementing a Search Index
39mAI-powered Semantic Search; A story of broken promises?