Architecting Scalable Multi-Modal Video Search
summary
An end-to-end system indexes petabytes by extracting tracked crops, embedding them and storing vectors for fast nearest-neighbor search; preprocessing uses YOLO V11 + bot sort for detection and tracking, on-GPU decoding and zero-copy tensor transfer to avoid CPU/GPU round trips, and key-frame hashing to deduplicate. Vector search uses HNSW-style indexes and a quadrant vector store, fine-tuned CLIP 2 and ArcFace for faces, and an agentic VLM stack (ven 3 8B + ven 2.57B); optimizations yield about a 10× throughput improvement on their prototype.
AI-written summary · about
chapters (12)
- 0:00Scalable video search intro
- 1:49Decard platform live demo
- 3:12Three-step pipeline
- 6:46Key frame selection & deduplication
- 7:06Semantic vector store with CLIP
- 9:16Fine-tuning CLIP embeddings
- 12:51AI agent for video intelligence
- 17:03Agent demo reasoning tools
- 18:30Prototype to production
- 23:07Zero-copy GPU acceleration
- 24:55Final takeaways
- 26:38Q&A
related talks
35mLarge-Scale Video Intelligence
50mSearching video content with AI
59mContextual search with vector search: exploring your options with open source tools
43mThe Search for a New Visual Search, Beyond Language
18mCoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics
52mBeyond Single Vectors: Multi-Vector Search for Enhanced...
43mOpenSearch and the Path to a Trillion-scale Vector Database
29mFrom Quick Wins to Revolutionising Productivity & CX with GenAI