Agentic AI Changed What We Test, And How We Test It
summary
Argues that testing must move from exact-match oracles to a bounded behaviour envelope with three gates: goal achieved, limits respected, and decisions grounded; evaluates agents across five dimensions (outcome, grounding, allowed tool sequence, safety/authorization, reliability). Describes AI-assisted test authoring, self-healing locators and the danger of a false green—example: checkout clicked "Save for Later" and passed—plus a media-player blueprint that ties timestamped trace evidence to a ship/no‑ship rule. Work in progress, not a finished case study.
AI-written summary · about
chapters (32)
- 0:00Agentic AI Changed What We Test, and How We Test It
- 1:07Speaker Intro - Pooja Oza, Senior QA Engineer, Walt Disney
- 2:12The Train vs the Autonomous Driver - Two Models of Testing
- 3:13What We Test Changed Before How We Test Changed
- 3:45From Exact-Match Oracles to a Bounded Behaviour Envelope
- 4:16Three Gates: Goal Achieved, Limits Respected, Decisions Grounded
- 4:47One User Goal, Three Valid Paths - All Should Pass
- 5:50Direct Execution, Asking for Clarification, or Escalating
- 6:22Fluency Is Not Correctness
- 6:53Five Evaluation Dimensions Instead of Pass or Fail
- 7:55A Ten-Second Quiz: Which Refund Answer Is Fabricated?
- 8:25Why the Confident, Concise Reply Was the Wrong One
- 8:55Real-World Failures With a Common Root Cause
- 9:26How We Test It: AI Fixes the Blank Page Problem
- 10:26Self-Healing Locators and the Danger of the False Green
- 10:57The Checkout Test That Clicked "Save for Later" and Passed
- 11:27Governed Healing: Prove the Outcome, Not Just the Selector
- 11:58Speed Is an Input, Not Release Approval
- 12:30Four Confidence Boosters and Which to Fund First
- 13:02A Blueprint: Agentic Testing for a Media Player
- 13:33Recap Markers, Intro Cues, Ad Breaks and Credits
- 14:36Connecting Trace Evidence to the Release Decision
- 15:37Four Explicit Proof Points Before a Build Is Shippable
- 16:07We Don't Ship When Tests Are Green - We Ship When Behaviour Is Bounded
- 16:37What This Means for Engineers, QA and Leadership
- 18:07The "Why Did We Ship" Story Leadership Actually Needs
- 19:08Three Questions to Ask on Your Very Next Pull Request
- 20:08Use AI for the Heavy Lifting, Keep the Keys
- 20:40Q&A: Capping Token Consumption in an Agentic Testing Loop
- 23:46Q&A: Measuring the Cost of a Business Outcome vs Cloud Resources
- 25:17Q&A: Which Skills Gap Hit the Team Hardest
- 27:20Q&A: Where Humans Still Clearly Beat Agents Today
related talks
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
43mTesting Agentic AI Applications: Beyond Traditional QA
41mHow AI is transforming software testing
44mHow AI is Exposing the Existential Crisis of Software Testing
47mHuman Experience Testing – Redefining Quality Engineering in the Age of AI
41mAI-Augmented Testing: How Generative AI, Prompt Eng Turn Testers Into Superheroes
28mAgentic AI
47mAgentic Testing: Your Skilled Human Tester