Hybrid search: Greater than the sum of its parts?
summary
Shows that combining BM25 and transformer embeddings in Vespa with an HNSW index and sentence-transformers (miniLM) yields better retrieval than either alone: dense gives ~94% recall@1000 in-domain while BM25 lags, but reciprocal rank fusion and simple linear blends raise scores on MS MARCO, Robust04 and TREC-COVID and help zero-shot domains. Notes embedding indexing (inner product), cross-/bi-encoders for re-ranking, learned sparse hybrids (ColBERTv2) and that the hybrid setup is practical and reproducible.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker background
- 3:11What is hybrid search?
- 9:25Traditional sparse retrieval
- 12:32Dense retrieval with transformers
- 18:28Comparing sparse and dense methods
- 24:51Combining sparse and dense retrieval
- 28:20Hybrid search in Vespa
- 36:46Why hybrid search works better
- 40:18Q&A session
related talks
42mJina Embeddings V2: From Raw Data to Bilingual Hybrid Search
37mLexical, Vector & Hybrid Search with Elasticsearch
25mSemantic Search
01h 30mBuilding a Search Engine
41mText to Context: How We Introduced Hybrid Search
24mRediscover Your Data with a New Multimodal Search Capability
42mResearch in the Age of AI
47mThe Neural Search Frontier #bbuzz