Mastering RAG Evaluation with MLflow & Open Source LLMs
summary
Shows a reproducible RAG evaluation pipeline that uses MLflow, LChain, and open‑source models hosted in AI Studio to generate and judge QA pairs, deploys an OpenAI‑compatible local proxy for MLflow, and logs traces/artifacts to the tracking server. Builds a golden set from 100 generated QA pairs and after LM judging and filters keeps 22 passed filters. Implements metrics for groundedness, robustness, faithfulness, correctness, latency, and retriever recall/ndcg, finds K>2 yields diminishing returns, and demonstrates working end‑to‑end results.
AI-written summary · about
chapters (10)
- 0:00RAG evaluation intro
- 4:14What is RAG evaluation
- 8:09Why MLflow for RAG
- 16:29Building evaluation dataset
- 25:46Filtering Q&A pairs
- 37:19Evaluating prompt templates
- 42:21Testing RAG pipeline
- 58:37Evaluating LLMs with prompts
- 1:09:43Measuring retriever performance
- 1:26:12Q&A
related talks
41mFrom Lab To Life: Practical AI System Evaluation
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
05mopen-rag-eval: RAG Evaluation without "golden" answers
10mModel innovation & evaluation with model partners Q&A
31mEvaluating RAG Applications: From Retrieval to Response Quality
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
32mEvaluating multi-turn conversations
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation