Decoding 1951 census: LLMs extract messy tables
summary
Digitizes India’s 1951 district handbooks across 40,000 pages by combining fine‑tuned OCR (Tesseract), layout detection (YOLO), and LLM orchestration, batching pages into 50 format types and processing left/middle/right pages sequentially with humans in the loop. Prototype uses Gemini 2.5 Pro and reaches about 90%+ cell-level accuracy on a ground‑truth test set, while validation at scale, cross‑page row alignment, and cost remain the main limitations.
AI-written summary · about
chapters (16)
- 0:00Digitizing 1951 census
- 2:49Messy historical data challenges
- 9:11Why OCR fails on tables
- 15:50LLMs for table variability
- 17:35Pre-processing setup
- 22:47Evaluating LLM output
- 26:17Limitations and validation
- 27:22Big picture impact
- 31:19Research goals
- 33:02Iterating on prompts
- 35:28Pipeline for 1961 census
- 36:31Extracting handbook text
- 37:34Digitizing maps
- 37:55Open source tools
- 39:01Model choices
- 39:22Q&A
related talks
01h 30mUnlock Information from Tables, Images and Complex Docs
01h 19mLearn to Unlock Document Intelligence with Open-Source AI
01h 30mPreparing Data for LLM pretraining using open source Data Prep Kit
17mFrom messy to meaningful data: LLM-powered classification in R
33mDesigning LLM Applications for Massive Datasets
26mReverse Engineering Using LLMs
15mHarnessing LLMs for Scalable Data Minimization
29mHow to roll your own LLM