Dense Vectors in Elasticsearch
summary
Experiment indexes ~600,000 news articles in Elasticsearch 7.x, adding distilbert sentence embeddings stored in dense_vector fields and using cosine similarity via script_score. Corpus index size grows from ~2 GB to 13 GB with vectors. Baseline BM25 / more_like_this on titles performs well and is fast; pure embedding search is slow and returns noisy topical jumps (Trump, sports, etc.). A manual BM25 + cosine blending recovers relevant sports results and adds useful diversity.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:00Background
- 2:20Application Context
- 5:50Embedding
- 7:40Performance Analysis
- 12:20Recap
- 13:50Questions
related talks
38mSearch: A New Era
19mVectorize Your Open Source Search Engine
01h 01mWord Embeddings - An Alternative & Efficient Approach to Search for Docs
26mImplementing a Search Index
47mLatest innovations from Elasticsearch: Drive speed, scale & relevance (PRT266)
45mRevolutionizing Search with Power of Neural Sparse and Dense Vectors
13mA Beginner’s Guide to Semantic Search
39mAI-powered Semantic Search; A story of broken promises?