Panel Discussion: What Can Go Wrong With AI in Testing?
summary
Panel covers concrete failures and experiments: AI agents deleting code, prompt injection and data leakage, Copilot-style edits that introduce fake functions and break working frameworks, and an experiment where asking for random animals reliably returns odd repeats like penguin axolotl capybara. They describe a workflow that extracts testable elements from 100 user stories producing a 70% interesting hit rate, and warn the productivity paradox where fast outputs cost hours to verify. Good if you want practical failure modes and mitigation tactics.
AI-written summary · about
chapters (21)
- 0:00AI testing panel intro
- 1:29Reputation risks in AI testing
- 3:56Cognitive offloading risks
- 5:20Trust erosion in AI outputs
- 7:33Accountability in AI reliance
- 8:36Worst AI testing failures
- 10:21Management cutting testers
- 12:06AI test code generation
- 14:40AI debugging pitfalls
- 16:49Productivity paradox in testing
- 21:11Bias in AI training data
- 24:25LLM hallucinations
- 27:54LLM failure patterns
- 29:39Ethical bugs from biased AI
- 32:11AI for test planning
- 36:04AI for code security
- 38:14AI as rubber duck tool
- 41:29AI speed vs startup limits
- 44:17Advice for testers on AI
- 47:13Why AI outputs fail tests
- 47:54Closing: human oversight
related talks
44mPanel Discussion | The AI Revolution in Testing: Opportunities and Challenges
41mPanel discussion: AI in code development – hype vs reality
46mThe Real Impact of AI in Software Testing
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
01h 05mPanel Discussion - The Future of Testing: Impact of AI in Quality Assurance and Beyond
01h 04mWhat are we thinking in the age of AI?
46mPanel Discussion: Beyond vibe coding: Expanding AI’s role in delivery
56mPANEL: Unleashing the Future: The Impact of Gen AI on Development Productivity