Panel Discussion: Who Tests the Machines? QE Leaders on Quality in the Age of AI-Written Code
summary
Panelists report concrete outcomes: startups ship nearly 90% AI-generated code, one org says 60–65%, another 40–50%; mobile automation built with Appium and MCP rises from zero to >80% coverage, but an initial batch of 200 tests yields 70% failing in week two until agents are taught the app. Teams run 25+ agents in production and push 300+ production fixes in three months, use Claude and CodeX, aim for ~70% agent accuracy, and shift QE work toward testing intent, risk scaffolding, and failure-triage agents.
AI-written summary · about
chapters (36)
- 0:00Who Tests the Machines? QE Leaders on Quality in the Age of AI
- 1:35Panel Intros - Ankur Agarwal, Nandini Srinivasan, Somu Suryanarayanan, and Rahul Nethi
- 4:07How Much Shipped Code Actually Involves AI Today?
- 4:38Close to 90% at a Startup - and Where the Remaining 10% Goes
- 5:40Code Generated in Half a Day, Reviewed for More Than One
- 6:41Firmware vs Software: Why the Numbers Differ
- 8:1640-50% and a Day-to-Day That Hasn't Fundamentally Changed Yet
- 9:19Delivery Speed and Instability Rising Together - Is It Real?
- 11:56Capturing AI Speed Without Sacrificing Stability
- 12:58Moving Away From Coverage Metrics Toward Risk Mitigation
- 14:34Fully Automated Means a Spike; Automated With Guardrails Doesn't
- 15:36You Don't Need an Engineering Product to See It - Try Claude on a Topic
- 17:08Why Subject Matter Expertise Is What Lets You Catch It
- 18:40A Real Failure: 200 Mobile Test Cases and 70% Failing in Week Two
- 19:41Teaching the MCP and the Agent About Your Own App First
- 20:43Reaching 70% Accuracy and Why 100% Is Still Hard
- 21:45Are We Debugging Intent More Than Code?
- 23:16Before AI a Failure Meant a Product Bug; Now You Check the Test First
- 24:18Building a Failure Analysis Agent to Triage the Triage
- 25:50Testing the Process Behind the Code, Not Just the Code
- 26:20The "Inactive User" Example: Why Specification Beats Syntax
- 27:54Shipping Production Fixes and Running Agents in Production
- 28:57The Lines Between Developer and Quality Engineer Have Blurred
- 29:3025+ Agents in Production and 300+ Production Fixes in Three Months
- 30:33The Posture That Matters: Product Depth, Not Tooling
- 31:04Don't Just Ask What Changed - Ask How and Why
- 31:35Specs and Tests Were Always the Asset; Customers Trump Both
- 32:37Shipping Speed May Outrun What Users Can Actually Adopt
- 33:39Being Competitive vs Building What Customers Actually Use
- 36:15What Skills Should an Early or Mid-Career Engineer Focus On?
- 36:45Domain as a Moat - and Why Code Still Matters
- 38:18Problem Solving and Systems Thinking Over Language Syntax
- 39:51Stop Learning How to Automate - Learn Risk and Scalability
- 41:58We Can Now Build the Wrong Thing Faster
- 43:01Teaching the Business Flow Before the Framework
- 44:02Software Development Got Cheaper; Software Engineering Did Not
related talks
47mPanel Discussion: QE Strategic Shift: What's Changing with AI, Automation, and Speed?
46mQA to QE: Scaling Quality with Ownership, Code, and Culture
42mIt's Inevitable, Mr. Anderson: Machines, Humans, and the Future of Testing
45mThe AI Revolution in Software Development: From Code Writing to Quality Assurance
22mBeyond Bugs: How AI turns QA into Full-Circle Engineers
59mPanel Discussion: Strategies for Scaling AI in Quality Engineering
47mEvolution of Quality Engineering in Financial Services
32mKeynote: AI & GenAI in Quality Engineering: Crawl, Walk, Run