Build Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
summary
Agents become the primary users of products, so public leaderboards mislead: the real question is whether an agent in this harness completes the job. The session shows how to build tasksets, seed environments, run 36 parallel sandboxes across SDK, CLI, MCP and custom agents, add trajectory-based rubrics, collect evidence and run repeated trials for significance. Live experiments on Sentry, Stripe and WorkOS reveal surprises — a vector DB test passes because the agent falls back to SQLite workaround and docs exceed context windows — and recommends classifying failures into actionable frictions.
AI-written summary · about
chapters (43)
- 0:00Start
- 0:05Session Opens - Build Your Own Benchmark
- 1:38Agents Are Becoming the Primary Users of Your Software
- 2:08Speaker Intro - Haritha Sreedharan Nair, Co-Founder and CTO at Oqoqo
- 3:09The Paradigm Shift: Who Is Actually Using Your Product
- 4:42MCP Servers, CLIs and Skills as the New Interaction Surface
- 6:13Why an Agent Is a Different Kind of User
- 7:15The New Question: Can This Agent, in This Harness, Do the Job?
- 7:46How Evals Evolved: From Spreadsheets of Prompts to Comparing Harnesses
- 9:48Vocabulary: Eval, Benchmark, Rubric, Trajectory
- 10:49The Task Is Not Just the Prompt
- 12:21A Real Example: Evals That Are Just Q&A Questions
- 13:22What Does Passing Really Mean?
- 13:53The Vector Database Experiment That Passed by Using SQLite Instead
- 14:53Public Benchmarks: What They Ask and What They're Good For
- 16:27A Good Leaderboard Can Still Fail Your Users
- 17:00Four Reasons Today's Evals Aren't Good Enough - Failure One: Too Simple
- 18:00Failure Two: The Eval That Reads Like a Tutorial
- 19:34Failure Three: Testing Against Only One or Two Agents
- 20:35Failure Four: Pass or Fail Hides Where the Product Broke
- 21:36Your Benchmark Is a Measure of Your Agent User Experience
- 22:08What a Good Eval Is: A Simulation of Real Work in One Unit
- 22:38Rule One: Don't Build Evals From Your API Surface
- 23:39Write the Eval From the User's Job Instead
- 24:11Rubrics Must Cover Output and Trajectory - and Stay Fair
- 24:43Run Repeated Trials for Statistical Significance
- 25:15Classify Failures So You Know What to Fix
- 25:45Why Companies Are Slow to Adopt: Fear of Cannibalising Their UI
- 26:46The Second Reason: There Is No Good Benchmark Infrastructure
- 27:17Live Demo: Experiments on Sentry, Stripe and WorkOS
- 28:19Building an Experiment: Tasks, Agents, MCPs and 36 Parallel Sandboxes
- 28:50Reading the Results: Steps, Tool Calls, Frictions, Token Cost
- 29:53The Sentry Finding: Docs Too Long for the Context Window
- 30:23Walking a Trajectory and Comparing Two Harnesses Side by Side
- 31:58The Takeaway: Agents Are a Real Product User
- 33:00Q&A: Testing an Agent's Recovery From Tool Failures and API Errors
- 34:00Q&A: Which Trajectory Details Matter and Which Are Noise
- 35:30Q&A: Handling Flaky Tool Calls and Third-Party Rate Limits
- 37:02Q&A: Where a Small Team Should Start
- 38:33Q&A: How Often Custom Benchmarks Need Updating
- 39:34Q&A: Permission Boundaries Without Exposing Real Credentials
- 40:35Q&A: Inefficient Success vs Failure With Better Reasoning
- 42:41Q&A: Testing Whether the Agent Picked the Right Tool or API
related talks
23mThe Art & Science of Benchmarking Agents
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications- 02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
01h 20mMind the Gap (In your Agent Observability)
20mAgentic Evaluations at Scale, For Everybody
37mAI agents testing How to evaluate the unpredictable
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
31mPanel Discussion: Beyond the Leaderboard...