Advancements in Evaluating Large Language Model Applications
summary
Explains concrete evaluation recipes: end-to-end business metrics for customer‑support summaries and reply suggestions, plus task‑level metrics (classification, retrieval, summarization). Uses RAG pipelines with vector stores, K‑NN retrieval, GPT‑3.5 for generation and embeddings, and the ragas framework to generate QA/context/answer triplets and compute answer‑relevance, context‑relevance and faithfulness. Combines human QA on 100–200 golden examples, regression tests, continuous monitoring via observability tools, and mitigations for LLM‑judge bias (swap inputs, repeat runs, ensemble scorers).
AI-written summary · about
chapters (9)
- 0:00Introduction to LLM evaluation
- 1:28LLM applications in production
- 5:27Evaluation levels and metrics
- 11:50LLM-as-a-judge approach
- 18:18RAG evaluation frameworks
- 25:47Transformer-based evaluation
- 31:29Observability tools for LLMs
- 33:17Demo: Evaluating RAG with Ragas
- 37:15Conclusion and key takeaways
related talks
14mImproving Large Language Model Performance
32mHolistic Evaluation of Large Language Models: From References to Human Judgment
35mEvaluating LLM systems: the nuts and bolts
34mEvaluating LLM Frameworks
01h 06mThe Future of Language Models: A Perspective on Evaluation
41mControlling Dependency on Third‑Party Large Language Models
28mNavigating Challenges and Enhancing Performance of LLM based Applications
30mEvaluating the evaluator- RAG eval libraries under the loop