Text Search on Images with Quantized ColPali
summary
ColPali is a late‑interaction vision‑LLM pipeline fine‑tuned on Pali Gemma that embeds whole page images so you can skip OCR, chunking and parsing; relevance scores compute per‑token similarities (max over document tokens, sum over query tokens). A quantized ColPali runs with ONNX inference and the embed-anything Rust tooling, reducing memory so local laptop search is feasible; real queries across many documents take about 23s without model upload and 34s with model upload. Current limits: PDF‑centric design and narrow multi‑vector support.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 1:26Challenges in simple RAG
- 3:34Problems with text extraction
- 5:00Introducing ColPali and its benefits
- 7:10How ColPali works
- 10:22Multi-vector retrieval explained
- 11:25Late interaction mechanism
- 12:48Quantization for memory efficiency
- 14:37ONNX inference and embeddings
- 16:01Performance and limitations
- 17:03Future plans and community
- 17:50Q&A session
related talks
33mUnlocking Insights From Multimodal PDFs Using OpenSearch and Vision-Language Models
52mBeyond Single Vectors: Multi-Vector Search for Enhanced...
11mPaliGemma – Making Gemma 2 see by adding a vision encoder
43mThe Search for a New Visual Search, Beyond Language
44mAI-Powered Search at Scale
01h 28mBuilding a Semantic Search Engine
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
32mEverything you wanted to know about VLMs but were afraid to ask