Testing Practices for Generative AI Systems: Focusing on Test Design and Analysis
summary
An in‑car system using generative AI (Gemini and Claude) for hundreds of millions of records and ~100k users requires strict output reliability, so the team avoids fully automated tests due to API cost and flakiness. They run Gemini and Claude in parallel with a cross‑reference check that only returns answers when models agree, focus testing metrics on outliers and precision, detect country‑specific biases, create example‑mapping prompts, and set release criteria based on daily user feedback rather than fixed benchmarks.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 12:49Testing criteria for GenAI
- 23:03Test design and analysis
- 31:56Prompt testing
- 41:23Summary
related talks
09mTesting Strategies for Generative AI Applications
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
46mGenerative AI: Say Goodbye to Deterministic Testing?
26mAI and the Future of Testing
31mTesting, Evaluating and Debugging Generative AI and Agentic Ap...
01h 04mWhat are we thinking in the age of AI?
37mGenerative AI: Beyond (Just) Hype
41mAI-Augmented Testing: How Generative AI, Prompt Eng Turn Testers Into Superheroes