From LLM-as-a-Judge To Human-in-the-Loop: Rethinking Evaluat...
summary
Adds LLM-as-a-judge support to OpenSearch 3.1 Search Relevance Workbench and ships a prototype RAGElo ELO-style framework that uses pairwise LLM comparisons to rank RAG agents without gold answers. Collects agent traces and answers into an OpenSearch index, runs pairwise judgments, produces ELO scores and textual LLM explanations of wins, surfaces hits and answer citations for human eyeballing, and finds correlation with experts but notes LLMs are overly optimistic and the UI and experiment-variant handling remain rough.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:13Evaluating GenAI challenges
- 7:10LLM-as-a-Judge limits
- 9:16Pairwise LLM comparisons
- 10:39ELO ranking system
- 15:18RAG-ELO demo
- 25:00Key learnings
- 27:50Q&A
related talks
49mLLM—Search Engine Killer?
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
39mRAGElo: An Elo Rating-based Evaluation Toolkit for RAG
32mA Practical Guide to LLM Evaluation
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
42mTesting What Matters: Evaluating LLM Relevancy with DeepEval
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
46mLLM—Search Engine Killer?