Scaling Trust, Not Automation - Rethinking Quality for the AI Era
summary
AI systems delegate judgment, so quality becomes measurable trust rather than more automation; real incidents include a 2016 Alexa clip that produced porn results leading to earcons from 2016 clip and a 2021 dangerous instruction, and early experiments with LLMs since GPT-2 produced a prototype automation framework with synthetic customers and judges. The math bites: two 95% components multiply, five → 77%, so the recommended approach is evaluation-driven design with inner component evals and an outer system loop, plus contract testing, observability, prompt/schema versioning, and a new QA scientist role.
AI-written summary · about
chapters (39)
- 0:00Scaling Trust, Not Automation
- 1:39Speaker Intro - Walter Zimerman, Senior SDE, Amazon
- 3:11Eight Years on Alexa Quality, From the Wrong Reason
- 3:42Confidence Is Not Correctness
- 4:13The 2016 Alexa Clip That Started an Internal Effort
- 4:45Earcons: When the System Stops Failing Loudly and Starts Failing Politely
- 5:45The 2021 Incident That Happened Anyway
- 6:47Experimenting With LLMs Since GPT-2 - and Why That Was Lucky
- 7:48Because Early Models Were Bad, He Never Had the Option to Trust Them
- 8:18Defining Terms: What Scale and Trust Actually Mean
- 9:48Scaling Trust Means Reliability That Compounds
- 10:19Traditional Tests Fail Loudly; AI Fails Confidently
- 10:50What Has Not Changed: The Application Still Has a Job to Do
- 11:52Pushing Back on "Traditional Testing Is Dead"
- 12:54What Changed Is Stochastic Components Inside the System
- 13:26The Volume Problem: 3,000 Failures and No Idea Which Are Real
- 14:27More Tests Do Not Mean Better Understanding
- 14:57Exhaustive Testing Is Still Impossible - Now Just More Expensive
- 15:28Testing vs Evals: Did It Pass, or How Good Is It?
- 16:29The Tension: Probabilistic Outputs, Deterministic Consumers
- 17:00What If the 1% That Fails Is Your Most Critical Path?
- 17:32An LLM Gives You a Beautiful, Confident, Completely Wrong Answer
- 18:03An Uncomfortable Story: A PoC That Found Real Bugs
- 18:33Finding Real Bugs in a Demo Is Not Being Trustworthy in Production
- 19:34The Math: Two 95% Components Do Not Make a 95% System
- 20:04Five Components Gets You 77%; Ten Gets You Below 60%
- 20:34Why You Can't Just Ship Fewer, Bigger Agents - Context Rot
- 21:38Measure From the Start: You Cannot Retrofit a Baseline
- 23:08Evaluation-Driven Design as the Analogue of TDD
- 23:39Write the Eval First, Watch It Fail, Build the Smallest Thing
- 24:41Double-Loop: Component-Level Inner, System-Level Outer
- 25:42Outer-Loop Failures Become Inner-Loop Tests
- 26:12Open Questions: Versioning Evals, Affordability, Emergent Bugs
- 26:44We Have Been Somewhere Like This Before - Microservices
- 27:46Contract Testing at the Seams Between Agents
- 28:16Observability Beat Testing in Production - and Will Again
- 28:46Schema Versioning: Changing a Prompt Changes a Contract
- 29:16Naming the Role: The QA Scientist
- 30:47Closing: You Have a Component Returning the Right Answer and Don't Know Why
related talks
16mTrust by Design: Building Reliable AI for Sensitive Environments
28mWhy Software Quality Is the Key to Scaling AI Development
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
23mHuman-Driven Intelligence: Where AI & Automation Fall Short
38mBuilding Responsible AI in the Age of GPT
57mToxic Trust: The Psychology of Automation Bias in Generative AI
44mToxic Trust: The Psychology of Automation Bias in Generative AI