Chatbot Arena: An Open Crowdsourced Platform for Human Feedback on LLMs
summary
A crowdsourced web platform runs pairwise anonymous battles between chatbots, records over 100K+ votes and millions of served requests, and converts pairwise wins into rankings using an Elo rating system. Results produce a public leaderboard comparing GPT‑4, Claude and many open models, and the collected data yields a 1M conversations dataset from ~200K users used to build moderation models and hard benchmarks. The serving stack and tools are released as FastChat open source.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:22LMSys team and goals
- 2:30Vicuna and open models
- 8:15Evaluating LLMs challenges
- 12:11Chatbot Arena concept
- 13:54Arena operation and data
- 16:43Arena impact and recognition
- 17:26Human feedback value
- 18:491M conversation data set
- 20:12FastChat open-source
- 20:55Future challenges
- 22:21Acknowledgments
- 23:26Q&A
related talks
32mChatbot Arena: UC Berkeley's Open AI Evaluation Platform
17mArchitecting LLM-Based Chatbots using free open-source LLMs
24mHow to Keep Your LLM Chatbots Real - A Metrics Survival Guide
26mAI Showdown: Open Source Tools for LLM Face-Offs
32mChatBot Learning Assistant with Large Language Models
47mCan You Trust Your ChatBot: Techniques for Testing LLM Responses
48mUnlocking LLM Potential: From Exploration to Integration
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid