Testing for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
summary
A pragmatic checklist for chatbot QA: nondeterministic LLM outputs and slow, rate-limited OpenAI calls break standard unit tests, so use lightweight assertions plus a fuzzy string diff matcher for tolerated differences, mock external calls and verify mock functions are invoked, lower temperature and lock prompts, and run semantic checks with DeepEval (relevancy, faithfulness, contextual precision) — one example run takes ~9s. Expect ongoing maintenance (models change, APIs deprecate) and avoid trying to run thousands (e.g. 3,000) live API tests in parallel.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 1:51Problem with AI chatbots
- 3:57Real-world AI failures
- 5:23Personal experience projects
- 6:26Key challenges in testing
- 8:54Deciding testing scope
- 9:57Technology adoption curve
- 12:03Basic testing approaches
- 13:29Using DeepEval for metrics
- 18:07Testing functions and prompts
- 20:35Key takeaways and advice
- 22:19Q&A session
related talks
42mRobust Testing Strategies for AI-Driven Applications
43mTesting Agentic AI Applications: Beyond Traditional QA
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
42mQA for ML: How we can trust AI with Food Sustainability
41mSurviving and Thriving with AI in QA
41mAutonomous Quality Engineering: AI Systems That Generate Execute Heal Learn and Govern Quality
13mNew Challenges in Software Quality Brought by GenAI and LLMs
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid