From Tables to Answers: building QA System for In-Document Searches
summary
Case study of a table QA pipeline that fine-tunes Tapas (base and large) on 30,000 annotations across ~3,000 tables to map queries to highlighted cells and simple aggregations; fine-tuned Tapas base is twice smaller and faster and doubles pre-trained scores though top accuracy sits near 60–65%. A binary LGBM classifier separates simple vs hard questions, retrieval succeeds ~80%, and the stack runs on a single A100 GPU with custom OCR/layout extraction, Elastic/Go search, MLflow REST serving; Llama 2 70B shows higher zero‑shot scores but slower inference and hallucinations.
AI-written summary · about
chapters (16)
- 0:00Table QA system intro
- 1:13Business problem overview
- 3:00Traditional search challenges
- 5:35State-of-the-art models
- 7:23TAPAS model selection
- 8:52Question types definition
- 10:37Data annotation process
- 11:41Evaluation metrics tests
- 13:08Training optimizations
- 14:56LLM approach with Llama 2
- 16:44LLM vs TAPAS comparison
- 17:29Confidence estimation
- 18:56Production pipeline
- 21:29Key takeaways
- 22:52Production evaluation
- 23:18Q&A
related talks
35mBuilding an enterprise NLSE ...
35mYou've got questions. We've got answers! #bbuzz
25mXavier Amatriain, VP of Engineering, Quora @ MLconf SEA
45mLessons learned in bringing a RAG chatbot with access to 50k+ diverse documents to production
48mPracticalities of Fine-Tuning Llama 2 with AI Studio
21mStop Making Models Bigger, Make Them Behave
01h 09mVector Search: Ask Me Anything!
42mAdvancements in Evaluating Large Language Model Applications