Document Vectors in the Wild
summary
Reuters builds a content-recommendation pipeline using doc2vec via Gensim trained on about 350,000 articles with 100‑dim vectors, clustering the document embeddings and assigning human labels to clusters; inference initializes a new document vector (stochastic) and converges quickly — typically <10 ms inference — and triplet tests give about 77% triplet accuracy. Production stack is a Python web app (Pyramid) on AWS with MySQL and Redis caching; A/B tests show the similar-article scroll significantly lifts engagement by a few percentage points.
AI-written summary · about
chapters (20)
- 0:00Introduction to the talk
- 2:30Speaker background
- 5:41Reuters and project context
- 6:25Infinite scroll motivation
- 8:12What are document vectors
- 9:59Creating document vectors
- 12:54Inference and stochasticity
- 14:18Clustering and labeling
- 16:04Testing model accuracy
- 18:55Measuring engagement impact
- 21:04Test branch results
- 23:11Surprising insights
- 24:36Inference randomness
- 26:22Training and caching speed
- 28:50Building the data science stack
- 32:02Data science platform features
- 35:36NER and trending topics
- 36:58Platform characterization
- 38:44Future of embeddings
- 41:11Closing remarks
related talks
50mVector Ops: run vector embedding-powered apps in prod
18m: Workforce Efficiency with AI-Driven Vector Search
26mPersonalizing User Content - The Power of a Vector & Search DB
45mA gentle introduction to Vector Databases
16mDense Vectors in Elasticsearch
34mWord Embeddings for fun and profit in Gensim
33mUsing the Amazing World
26mBeyond the Vector DB