Chatbot testing in the real world: Making sense of an emerging practice
summary
Claims that chatbot QA is about testing experience, not just correctness and that a custom NLU testing framework catches intent drift early; describes running thousands of real utterances in parallel, pulling detected intent and confidence via OData queries and failing CI when thresholds drop. It also uses a regex PI redaction mechanism to strip sensitive tokens before analytics. Examples include detecting overlapping intents after adding hotel flows and reducing false routing by tuning confidence thresholds.
AI-written summary · about
chapters (9)
- 0:00Introduction to chatbots
- 1:37Chatbot testing complexity
- 3:41Chatbot architecture
- 6:21Team collaboration
- 7:43Defining testing experience
- 8:53Testing challenges
- 12:49Testing solutions
- 16:17Lessons and best practices
- 18:35Future trends and conclusion
related talks
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
47mTesting Conversational AI
17mChaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents
43mTesting Agentic AI Applications: Beyond Traditional QA
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
54mIntroduction to Chatbots
42mTesting Conversational AI - Strategy to Automation
13mExploring AI-Powered Chatbots and Their Transformative Role