Turning Fails into Features: Zapier’s Hard-Won Eval Lessons
summary
Zapier Agents turns failures into a data flywheel by instrumenting traces (tool calls, errors, pre/postprocessing) and offering a one‑click path to convert a real run into an eval; explicit thumbs‑up placement noticeably increases feedback, implicit signals (turning an agent on, copying responses, curses) feed weekly Slack reports, and LM judges (Braintrust/MCP) explain root causes — e.g., Gemini Pro “yaps” vs Claude’s decisive execution and bad JSON. They run trajectory evals that can take up to an hour, keep regression and aspirational sets (aiming for things like 200 tool calls), and validate changes with 5% AB rollouts.
AI-written summary · about
chapters (6)
- 0:00Introduction to Zapier agents
- 1:46Building probabilistic AI agents
- 3:30Collecting actionable feedback
- 6:41Understanding agent runs at scale
- 9:07Evaluating with different eval types
- 14:47Closing thoughts on evals and users
related talks
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
35mThe Agentic AI Engineer- 02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
47mHow to Build Enterprise-Grade AI Agents Using Robust Evaluation
01h 26mShipping AI That Works: An Evaluation Framework for PMs
01h 20mMind the Gap (In your Agent Observability)
19mYour Biggest User Isn't Human: Designing DevTools for AI Agents