Behind the hype: managing billion-scale embeddings in Elasticsearch and OpenSearch
summary
Managing 1 billion embeddings in Elasticsearch and OpenSearch requires concrete tradeoffs: GPU + NVIDIA Cagra speeds up index building but a naive float32 ANN needs ~4 TB RAM while raw storage is ~4 TB on disk; quantization + reranking cuts RAM to hundreds of GB (scalar int8, product or binary PQ) with recall loss recovered by two‑phase rerank; merge segments to a single HNSW/IVF graph for faster search; consider disk BBQ / vectors-on-disk and measure recall (MRR/NDCG) on a test set before full indexation.
AI-written summary · about
chapters (15)
- 0:00Billion-scale embeddings intro
- 5:13Why one billion vectors fail
- 7:22Chunking text for embeddings
- 10:29Indexation challenges at scale
- 14:19GPU acceleration for vectors
- 16:48Disk needs for billion vectors
- 17:31Avoid duplicate vector storage
- 21:50Efficient embedding search
- 24:42Approximate nearest neighbor
- 27:52Lucene filtering improvements
- 30:19Merging segments faster
- 31:42Quantization for memory
- 35:56Vector compression trade-offs
- 38:25Beyond quantization solutions
- 40:11Key takeaways and future
related talks
38mSearch: A New Era
48mUnlocking Next-gen Semantic Search With OpenSearch 3.x
34mThe Challenges of Making Vector Search Billion-scale
36mWhen Semantic Search Breaks in your RAG systems: RAM Walls, Silent Failures, and the Architecture Decisions
32mRetrieval Augmentation and Semantic Search at Scale
47mBuilding intelligent analytics at scale
22mVector Search Made Simple: Getting Started With OpenSearch for AI Applications
01h 09mVector Search: Ask Me Anything!