Agentic Evaluations at Scale, For Everybody
summary
Claims that matter: SWE-Bench Pro shows frontier models within a few percentage points while the harness shifts performance by 22%; a lab can rerun a Kaggle benchmark with different compaction and publish much better but incomparable results. Built and running: an open benchmark platform on Kaggle, a PvP Game Arena with unsaturating Elo ratings, a standardized agent exam that returns 500+ submissions in week one. Engineering details: Bradley‑Terry pairing, ~400,000 poker hands for significance, Colab LLM proxy and open‑source datasets.
AI-written summary · about
chapters (11)
- 0:00Agentic evaluations intro
- 1:59Why AI evals are broken
- 4:07Need for open evaluations
- 5:52Kaggle’s solutions
- 7:39Hackathons for benchmarking
- 9:47Standardized agent exams
- 12:12Game Arena benchmark
- 15:22Scalable benchmarking challenges
- 16:45Kaggle Benchmarks platform
- 18:29Sustaining open benchmarks
- 19:31Agentic benchmarking challenges
related talks
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
11mThe Unbearable Lightness of (Agentic) Evaluations
19mAI's Infinite Game
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
19mFuzzing in the GenAI Era