Reducing Hallucinations and Evaluating LLMs for Production
summary
Shows an actionable pipeline: map docs into vector DBs, build a RAG or LangChain stack, create a manual evaluation set of hundreds of prompts, then automate checks with Deepchecks and the OpenLM Leaderboard / Eleuther AI harness. Diagnoses specific hallucinations—fabricated facts, prompt contradictions, context conflicts—and reduces them with input templates, context injection, fine‑tuning, RLHF and LM‑based evaluators that score bias, toxicity and factuality. Emphasizes continuous validation in production and tradeoffs between heuristic, LM‑based and costly human evaluation.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:10LLM evaluation challenges
- 5:03Understanding hallucinations
- 10:41LLM evaluation for production
- 13:10Challenges with LLM evaluation
- 16:00Open LLM leaderboards limits
- 17:49Key LLM benchmarks
- 19:33Custom LLM evaluation sets
- 21:00LLM evaluation techniques
- 22:46Mitigating LLM hallucinations
- 23:51LLMs production best practices
- 24:54LLM evaluation tools
related talks
39mLLMs and Hallucinations: Understand, Measure, Control
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
32mWhy do LLMs hallucinate?
38mMaking LLMs reliable - A practical framework
32mEvaluating multi-turn conversations
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
46mEnhancing Agent Reliability Through Evaluation-Driven Development
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications