Latest innovations from Elasticsearch: Drive speed, scale & relevance (PRT266)
summary
Elastic adds searchable vector fields using 768‑dimension BERT vectors loaded from Hugging Face via the Eland utility, querying with a KNN clause and an HNSW index for nearest‑neighbor search. Hybrid scoring mixes BM25 and vector boosts. Examples: date histograms run 8x faster, terms aggregations ~20% faster, random‑sampler aggregation can be up to 80x faster for large counts. Storage shows a worked example shrinking 100 TB to ~30 TB (≈70% savings) using match‑only fields, doc_values, synthetic source and time‑series routing; frozen/searchable snapshots enable large cold archives.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:20Vector search lets you find what you mean, with higher relevance
- 2:16Convert data into vector representation
- 6:31Apply vector search in 3 steps
- 10:37Vectorize during data ingestion
- 11:35Find nearest neighbor, approximately
- 12:40Hybrid scoring gets you the best of both worlds
- 13:52Approximate nearest neighbors scale vector search
- 15:03How will you apply vector search?
- 16:54NLP #1: No more browsing manuals/FAQs
- 18:41NLP #2: Prevent customer service escalations
- 20:34Personalization is a "data" problem
- 23:29Crawl or connect searchable data
- 24:00Pipeline management for ML models
- 24:26Topics we don't have time to cover
- 28:32Storage savings
- 32:01Memory savings
- 34:48Performance
- 43:00Serverless Elasticsearch
- 45:52Community resources
related talks
38mSearch: A New Era
01h 09mVector Search: Ask Me Anything!
30mExtracting insights from unstructured data
42mResearch in the Age of AI
15mMastering RAG and Vector Search with Elastic
24mVector Search at Uber
37mLexical, Vector & Hybrid Search with Elasticsearch
45mBehind the hype: managing billion-scale embeddings in Elasticsearch and OpenSearch