Is This Actually Getting Better? A Three-Layer Approach to Measuring AI Products
summary
Case study of an AI coding agent that actually measures results rather than promising examples: a pragmatic three layers: offline, online, qualitative evaluation, a bespoke human‑curated dataset, and LLM as a judge for code quality. Offline shows strengths on bug fixes, unit tests and docs but not new features; online reveals setup friction and power users; qualitative interviews surface UX fixes. Reported outcomes include 20% more PRs and 2–3 hours saved per developer per week, plus concrete trade‑offs and launch timing.
AI-written summary · about
chapters (16)
- 0:00Welcome
- 1:51Why measurement matters
- 4:19Three layers of AI quality
- 6:46Challenges in LLM output
- 9:16Offline evaluation
- 11:22Online user metrics
- 12:48Qualitative user feedback
- 14:13Rodev AI coding agent
- 16:00Defining coding problem
- 18:08LLM as code judge
- 19:54Building offline dataset
- 21:00User feedback insights
- 23:30Trade-offs for release
- 25:35Key takeaways
- 27:03AI getting better?
- 28:05Q&A
related talks
19mBuilding AI Products That Actually Work
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
15mPractical tactics to build reliable AI apps
27mBuilding AI applications: Successes and Lessons
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
32mMeasuring Developer Experience in the Age of AI
22mThe AI Productivity Paradox in Engineering
33mEvaluating Your AI Agent