James Bach Shares His Opinion on #SoftwareTesting and #ChatGPT
summary
Argues that ChatGPT cannot do reliable testing and demonstrates it with Bing with GPT‑4 by asking it to summarize bach.geophysicsassistant.com and watching it hallucinate a non‑existent product and cite irrelevant sites. Feeds a real social‑media user story about photo tagging and gets shallow, name‑filled test cases, no negative coverage and no questions. Reports experiments with local, open source and API setups, lists 15 syndromes (incurious by design, hallucination, incongruent answers, numeric errors, opacity, unteachable behavior) that make it unsuitable for serious testing.
AI-written summary · about
chapters (5)
- 0:00The hype vs reality
- 1:14Hallucinations and unreliability
- 5:13Analyzing AI-generated test cases
- 9:52The role of a competent tester
- 10:52Fundamental limitations of LLMs
related talks
01h 04mWhat are we thinking in the age of AI?
01minteresting experience with ChatGPT
39mHow ChatGPT helps to boost your development efficiency
32m🧐 A Look at Possibilities of Using AI in #SoftwareTesting
31mGetting the most out of Chat GPT
01h 16mChatGPT - and How we Can Benefit from it
25mChatGPT Unleashed: From Code Wizards To Silly Shenanigans!
41mHow you can trick your company into hiring ChatGPT