Evaluating RAGs: On the correctness and coherence of Open Source eval metrics
summary
Evaluates open‑source RAG eval libraries (RAGAS TruLens and reuse) by running a 200‑question benchmark with 5 retrieved chunks per query using ChromaDB and generation LLMs (Llama variants Claude Titan GPT‑3.5/4). Metrics use LLM extractors; evaluator LLMs disagree widely: 48% zero overlap in retrieval, many evaluators still report nonzero precision/recall, and faithfulness can be nonzero with irrelevant context. Notable coherence only within GPT‑3.5 and GPT‑4 family. Practical takeaway: automated metrics run but frequently disagree and sometimes return nulls or misleading scores.
AI-written summary · about
chapters (19)
- 0:00Welcome
- 0:43Presenters overview
- 1:25What is RAG
- 2:08How RAG works
- 3:52Why RAG evaluation matters
- 5:15Open source RAG eval libraries
- 6:18Retrieval evaluation metrics
- 8:27Context recall and precision
- 11:16Generation evaluation
- 13:23LLM evaluation problems
- 14:28Evaluator experiment setup
- 15:53Evaluation results
- 17:59Judge LLM agreement
- 20:05Context precision and recall
- 21:53Faithfulness challenges
- 24:21Coherence in model families
- 25:47Evaluation technical challenges
- 27:13Key takeaways
- 27:55Q&A
related talks
30mEvaluating the evaluator- RAG eval libraries under the loop
22mCracking the Code: Deciphering Evaluation Essentials for RAG
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
05mopen-rag-eval: RAG Evaluation without "golden" answers
40mHow [not] to evaluate your RAG
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You