Own the loop: how designers and researchers win with AI evals
summary
A concrete claim: designers and researchers must own AI evaluation by defining what “good” means, writing rubrics, calibrating graders and feeding those signals into models. Evidence includes a rehab pilot that used wearable sensors and an app where a technically accurate model misled a patient, prompting a redesign; a Figma loop where grader calibration raises scores; and a shift from manual grading to LLM-driven scaling to thousands via auto evals, complemented by human evals and UX evals and four levers: prompts, eval design, datasets, capabilities.
AI-written summary · about
chapters (13)
- 0:00AI evals in design intro
- 1:05Neurosurgery to AI healthcare
- 4:58Why AI evals matter
- 7:04What an eval is
- 8:47From verification to orchestration
- 10:58Edna Mode and AI loop
- 11:47Three approaches to steering models
- 13:30UX evals close gaps
- 14:52Researchers lead AI evals
- 15:54Measuring model quality
- 16:58Shaping AI products
- 17:19Four levers for AI behavior
- 18:00Own the loop
related talks
24mHow to Test AI Features Before Your Users Do
01h 26mShipping AI That Works: An Evaluation Framework for PMs
31mBeyond Vibes: Building Evals for Generative AI
30mIs This Actually Getting Better? A Three-Layer Approach to Measuring AI Products
19mBuilding AI Products That Actually Work
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
34mScaling Trust, Not Automation - Rethinking Quality for the AI Era
26m7 Habits of Highly Effective Generative AI Evaluations