Lies, damned lies and large language models
summary
Demo measures LLM hallucinations by running 817 TruthfulQA questions through models and Python tooling, using TruthfulQA and Hugging Face datasets and transformers plus langchain to call GPT‑3.5‑turbo. Results show a baseline around 36% correct in one run, and cited work raises accuracy from ~70% to 84% with a retrieval strategy. The presentation details prompting, fine‑tuning, self‑refinement and retrieval augmented generation and warns where RAG pipelines commonly fail.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:03What are large language model hallucinations
- 3:08How LLMs learn and hallucinate
- 7:23Data sources impact
- 9:51Base models vs apps
- 10:34Classifying hallucinations
- 12:19Measuring hallucinations
- 14:26Demo: TruthfulQA
- 19:19Hallucination challenges
- 20:21Mitigating hallucinations in LLMs
- 22:48Retrieval Augmented Generation
- 24:55RAG pipeline pitfalls
- 27:03Advanced RAG methods
- 28:28Resources
related talks
41mTruth Checker: Generative Large Language Models and Hallucinations
43mBeyond the Hype: A Realistic Look at Large Language Models
01h 04mCan you trust your (large language) model?
32mWhy do LLMs hallucinate?
16mNavigating the Uncharted: Ensuring Prompt Quality in the Age of Language Models
51mLarge Language Models: Friend, Foe, or Otherwise
24mResponsible AI: Assessing the Truthfulness of AI Model Responses
34mBuilding Trust with LLMs