Semantic Multimodal Multilingual Similarity Engine
summary
A working semantic multimodal multilingual similarity engine is described with a bias toward GPU‑accelerated inference and distributed deployment, built on a Vector database that stores precomputed embeddings. It uses CLIP for image–text dual encoders, sentence Transformers for text, and indexes via HNSW with quantization often served through Faiss. Training uses large scraped corpora (LION dataset up to 5 billion pairs, a ~3 TB pretraining chunk) and production optimizations: freeze norm layers, low learning rates, quantization and ONNX/TensorRT runtimes; results are "close to good enough."
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:16Semantic search importance
- 3:04Lexical search limits
- 5:35Semantic search design
- 7:02Encoding text and modalities
- 9:57Efficient retrieval indexing
- 11:24Multilingual models
- 13:32CLIP for multimodal
- 17:06CLIP working tips
- 18:33Beyond text and images
- 21:47Semantic search production
- 23:13Vector databases
- 23:59Q&A
- 26:10Q&A
- 33:03Q&A
- 36:18Q&A
- 39:10Q&A
related talks
56mApplications of Multimodal Learning in Media Search Engines
25mSemantic Search
22mMultilingual embeddings to scale NLP models to multiple...
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
13mTraining Embedding Model Resiliently for Multimodal M...
32mGenerative AI Text and Multimodal Embedding Models for Real World Use Cases
57mUsing Vector Databases for Multimodal Embeddings and Search
22mBuilding a Multimodal Search Engine