Rediscover Your Data with a New Multimodal Search Capability
summary
Claim: adding multimodal embeddings to OpenSearch via AutoGluon yields consistent gains over BM25. They benchmark four datasets (biomedical, medical, finance, QA) using DCG@10 and show a hybrid BM25+vector pipeline (BM25 top‑100 then KNN on embeddings) wins almost 100% of the time, and fine‑tuning embeddings on relevance data gives the best scores. Tooling includes Hugging Face/CLIP backends, auto‑ensembling with LightGBM/XGBoost, three‑line training, and an embedding extractor in AutoGluon master ready for OpenSearch integration.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:24Lexical Search: Document Retrieval
- 2:51Semantic Search: Find the Meaning
- 3:50Real data is messy
- 6:00Two Tower Architectures
- 7:43Semantic Retrieval with Deep Learning
- 8:24Pretrained embeddings
- 10:14Pretrained Embedding Models or BM25?
- 14:50Improve the Pretrained Embedding Models
- 17:47Multimodal DataFrame
- 19:19Three-lines of Code
- 22:57Roadmap
related talks
41mText to Context: How We Introduced Hybrid Search
26mAdvancing Multi-Modal Search Capabilities in Search Pipeline
45mHybrid search: Greater than the sum of its parts?
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
39mAI-powered Semantic Search; A story of broken promises?
46mNatural Language & Search, Improve your Search w/ Vector capabilities
25mSemantic Search
48mUnlocking Next-gen Semantic Search With OpenSearch 3.x