The Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
summary
Dashboards report Cosine Similarity, BLEU, BERTScore and show “90% accurate” while generated answers flip legal roles — nominated beneficiary becomes legal heir — or confidently hallucinate despite Context Recall 85% and Answer Relevance 92%. The fix is concrete: decompose retrieval vs generation, run adversarial tests (negation injection, entity swapping, multi‑hop, retrieval-gap cases), use LLM‑as‑a‑Judge with G‑Eval and ensemble or cross‑model judging to kill self‑preference, and evaluate with Ragas and DeepEval while using prompt caching and hierarchical judges to control cost.
AI-written summary · about
chapters (14)
- 0:00RAG evaluation pitfalls
- 2:11RAG metrics failures
- 6:44Why NLP metrics fail RAG
- 8:10RAG: retrieval vs generation
- 10:20Faithfulness and relevance metrics
- 13:10LLM as semantic judge
- 15:41Point-wise vs pairwise LLM eval
- 18:46Bias in LLM judges
- 21:59Limitations of test sets
- 23:00Adversarial test sets
- 26:34Maintaining eval data
- 27:41Ragas and DeepEval
- 29:06Cost-effective eval
- 31:34Key takeaways
related talks
30mEvaluating the evaluator- RAG eval libraries under the loop
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
31mEvaluating RAG Applications: From Retrieval to Response Quality
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
05mopen-rag-eval: RAG Evaluation without "golden" answers
31mBeyond LLMs and Maturing RAG use cases