Applications of Multimodal Learning in Media Search Engines
summary
Describes a production media search system that trains a single joint vector space for queries, tags, and GIFs from billions of anonymized events using Word2Vec/skip-gram, StarSpace and contrastive training with BERT-style text encoders and multimodal fusion. It serves sub-millisecond nearest-neighbor retrieval via nmslib KNN indexes, integrates vectors into Elasticsearch through query expansion and tag fields, and reports A/B lifts (~10% on search metrics, ~40% on recommendation CTR); cold-start is handled with n-grams, metadata and multimodal encoders.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 11:18Multimodal vector representations
- 20:18Vector implementation in GIF platform
- 32:43Evolution and cold start issues
- 46:25Summary
- 48:57Q&A
related talks
57mUsing Vector Databases for Multimodal Embeddings and Search
41mSemantic Multimodal Multilingual Similarity Engine
22mBuilding a Multimodal Search Engine
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
26mAdvancing Multi-Modal Search Capabilities in Search Pipeline
42mResearch in the Age of AI
24mRediscover Your Data with a New Multimodal Search Capability
32mGenerative AI Text and Multimodal Embedding Models for Real World Use Cases