Supercharge Information Extraction with Vision-Based LLMs
summary
Claims automation for an insurer that replaces heavy OCR + regex and custom training with Vision LLMs plus an OCR pre-step and prompt engineering, implemented as an event-driven pipeline that outputs strict JSON and compares results to existing ground truth. Uses AWS Textract for OCR, CLIP-style image-text embeddings, and iterative prompt tuning; initial runs hit 60–85% accuracy and team measures per-field, overall-field and perfect-document accuracy while noting cost and latency are lower than custom-model routes and improvements show diminishing returns.
AI-written summary · about
chapters (8)
- 0:00Vision-based LLMs intro
- 1:25Information extraction importance
- 3:12Technical extraction evolution
- 8:06Vision LLMs game changer
- 10:35How vision LLMs work
- 15:42Claims automation case study
- 23:07Key insights from projects
- 27:20Q&A
related talks
32mDocuments Meet LLMs: Tales from the Trenches
42mUsing Deep Learning to Understand Documents
42mExtract information accurately from documents using computer vision
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
32mRevolutionizing Automation: Retrieval-Augmentation with LLMs
37mVisual Literacy: Complex Document Retrieval with VLMs
33mDeep research at scale
35mGenerative AI for Fraud Detection & Claims Analysis