VerifAI: Biomedical QA with Verifiable Answers
summary
VerifAI is an open-source biomedical QA pipeline that combines lexical OpenSearch and semantic Qdrant retrieval over ~35 million PubMed abstracts, uses a hybrid weight of 0.7 lexical / 0.3 semantic, and retrieves top-10 abstracts into a prompt. Generation uses Mistral 7B fine-tuned (LoRA) on a 9,000-question pqi_ref dataset (27M trainable params, 2 epochs, ~32 hours), and verification is a BERT-based entailment classifier with ~90% F1 that flags unsupported sentences. Quantizing Qdrant vectors from 32-bit to 8-bit cuts search latency from ~40–50s to ~0.5s. All code, models and data are available open source.
AI-written summary · about
chapters (13)
- 0:00VerifAI project intro
- 2:59LLM limits in biomedicine
- 4:43Biomedical QA challenges
- 6:06VerifAI system overview
- 10:02Biomedical literature retrieval
- 15:47Generative component & fine-tuning
- 17:37Verification engine
- 19:25Evaluation results
- 24:04User interface demo
- 26:18Perplexity comparison
- 27:43Open-source deployment
- 29:49Future developments
- 30:10Acknowledgments
related talks
43mAdvanced Retrieval-Augmented Generation Techniques
30mResponsible AI with fmeval - an open source library to evaluate LLMs
40mDeploying Your Own Language Models: A Use Case Demo with Open Source Tools
32mEvaluation of QA systems built on LlamaIndex
46mLive coding a quick & easy semantic search with Qdrant from scratch
43mHarnessing AI to strengthen trustworthy information
35mCan AI solve the biggest problem in drug discovery? Time!
42mQA for ML: How we can trust AI with Food Sustainability