Cracking the Code: Deciphering Evaluation Essentials for RAG
summary
RAG accounts for 65–70% of generative AI use cases and evaluation focuses on precision, relevance, faithfulness, groundedness, coherence, summarization, safety and latency; Phoenix offers per-query tracing and explainability, Athena exposes ~40+ metrics, Ragas publishes four core metrics, and Aries/others are frameworks under review. Most platforms use LLM-as-judge, incur token cost, and lack automated data preprocessing checks (no chunk-size/overlap diagnostics); only one framework supports built-in human evaluation. Quadrant is a Rust-based vector store with quantization for performance and is presented as production-ready.
AI-written summary · about
chapters (8)
- 0:00Introduction and welcome
- 1:03What is RAG evaluation
- 3:27Knowledge base in RAG systems
- 4:51Why evaluation matters
- 8:22Choosing evaluation metrics
- 10:06Meta comparison of frameworks
- 15:30Limitations of current frameworks
- 16:36Q&A session
related talks
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
40mHow [not] to evaluate your RAG
31mEvaluating RAG Applications: From Retrieval to Response Quality
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
14mBeyond RAG: Building Reliable AI Systems for Privacy Assessments
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
05mopen-rag-eval: RAG Evaluation without "golden" answers
34mEvaluating LLM Frameworks