Unlocking Insights From Multimodal PDFs Using OpenSearch and Vision-Language Models
summary
Shows two concrete pipelines: one converts PDF tables/images into text (CSV, captions) and stores Titan embeddings via OpenSearch ingest pipelines; the other indexes page images with multi-vector VLMs like ColPali late interaction and rescoring. Demo indexes 7,000+ pages, returns exact table-derived answers (London highest price) and exposes token→patch attributions. Claims pooling to 3 vectors keeps 99.7% retrieval quality, offers an ML connector to SageMaker and an open Hugging Face demo; RFC for native late‑interaction support is filed.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:27Retrieval Augmented Generation in OpenSearch
- 3:56Multimodal PDF challenges
- 7:30Grounding multimodality into text
- 12:51Vision language models for retrieval
- 21:58Multi-vector retrieval best practices
- 26:02Choosing the right approach
- 27:47Q&A
related talks
26mAdvancing Multi-Modal Search Capabilities in Search Pipeline
36mColPali’s Vision-Powered RAG for Enterprise Documents
01h 00mComplex Data Ingestion with Open Source AI
01h 25mLearn to Unlock Document Intelligence with Open-Source AI
01h 24mVoiceVision RAG - Integrating Visual Document Intelligence with Voice Response
40mKeynote: Open-source Multimodal AI in the Wild
01h 19mLearn to Unlock Document Intelligence with Open-Source AI
52mBeyond Single Vectors: Multi-Vector Search for Enhanced...