AI Showdown: Open Source Tools for LLM Face-Offs
summary
An open source workflow called LLM Comparator provides a Python package (pip installable) plus an interactive UI that runs side-by-side comparisons, produces a single comparator.json replay artifact, and outputs per-prompt decisions with free-text rationales that are clustered and visualized; it supports live API pulls or local models, uses judge ensembles and a flip position to reduce bias, reports win rates and slices, and currently compares two responses at a time; alternatives: LM Evaluation Harness, OpenAI Evals, Hugging Face Evaluate.
AI-written summary · about
chapters (15)
- 0:00Welcome
- 1:47Structured LLM evaluation
- 3:57Traditional vs modern evaluation
- 6:47Open-source tools for trustworthy AI
- 7:49Introducing LLM Comparator
- 10:21How LLM Comparator works
- 13:55Processing judge rationales
- 15:23Understanding comparator.json
- 16:26Visualizing results
- 17:31Live demo of LLM Comparator
- 18:56Other open-source tools
- 20:21Choosing the right tool
- 21:02Best practices for evaluation
- 22:27Future trends in LLM evaluation
- 23:11Resources and closing
related talks
33mFrom LLM-as-a-Judge To Human-in-the-Loop: Rethinking Evaluat...
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
32mChatbot Arena: UC Berkeley's Open AI Evaluation Platform
35mOpen-source AI: why it matters and how to get started
30mResponsible AI with fmeval - an open source library to evaluate LLMs
27mChatbot Arena: An Open Crowdsourced Platform for Human Feedback on LLMs
15mOpenAssistant's Opensource LLMs Saga
01h 28mMastering RAG Evaluation with MLflow & Open Source LLMs