Evals Are Not Unit Tests
summary
Built a Fruit Letter Counter on V0 using the AI SDK and GitHub sync, hits GPT‑4.1 variability: it returns the right answer 10/10 then later fails a real user’s long query. Claims 95% of behavior is stable and 5% causes outages. Proposes basketball court evals that map user queries, collect thumbs up/down and logs, factor constants in data, variables in task, score deterministically (pass/fail), and run evals in CI to catch regressions before deploy.
AI-written summary · about
chapters (10)
- 0:00Introduction to Vercel's V0 and its growth
- 1:00The problem with AI unreliability
- 2:44The "Fruit Letter Counter" app example of AI failure
- 3:33Introducing "evals" and the basketball court analogy
- 5:09Defining the "court": understanding the domain of user queries
- 7:53Data collection for evals
- 9:13Structuring evals: constants in data, variables in task
- 10:45Scoring evals
- 12:35Integrating evals into CI/CD
- 13:40The benefits of using evals
related talks
24mHow to Test AI Features Before Your Users Do
13mHow to evaluate your Evals?
47mBuild Trustworthy AI Agents powered by Evals
38mBuild Evals That Actually Matter
31mBeyond Vibes: Building Evals for Generative AI
46mEnhancing Agent Reliability Through Evaluation-Driven Development
38mAI Just Happened to Me : Help!!
37mAI agents testing How to evaluate the unpredictable