Natural Language Search-Lexical and Semantic
summary
Covers how OpenSearch moves from term matching to meaning by combining BM25 scoring with sparse vs dense embeddings, LLM-generated vectors and neural + kNN plugins for indexing and query-time embedding. Describes chunking, ingest pipelines, hybrid scoring and normalization, reports a 10–177% uplift versus BM25 and ~14% vs dense on the speaker’s test, and points to a live demo at https://ml-search-331750873.us-east-1.elb.amazonaws.com/ and real-world scale examples (100M songs / 1B vectors at 7k qps).
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:49Search history and evolution
- 4:18Lexical search basics
- 9:01Ranking with BM25
- 10:07Limitations of lexical search
- 10:28Vector embeddings in search
- 13:45Dense vs sparse vectors
- 16:14ML and LLM terms
- 17:17Vectors to search methods
- 19:04User behavior in relevance
- 20:10Hybrid search architecture
- 21:58Building hybrid pipelines
- 22:18Sparse vector performance
- 24:34Hybrid search scoring
- 25:38Multimodal search
- 26:19Vector search applications
- 27:21Q&A
- 27:44Search vs ML roles
- 29:52Semantic knowledge in logic
- 34:48Vector search deployment
related talks
51mQuery Understanding With Large Language Models: Techniques and...
47mBoosting AI With OpenSearch: Vector Stores, Similarity Search, and Retrieval-Augme...
44mRediscover your keyword search: Expand, Enrich and Rewrite
30mBeyond Keywords: Building an AI-Powered Search That Reads Your Mind
45mRevolutionizing Search with Power of Neural Sparse and Dense Vectors
22mAgentic Hybrid Retrieval: Orchestrating LLM Agents, Hybrid Search & Explainability in OpenSearch
21mBuilding RAG With OpenSearch ML Plugins: From PDFs To Voice-Enabled Search
39mExperiments with OpenSearch and AI