New Challenges in Software Quality Brought by GenAI and LLMs
summary
Users now expect chunked, near‑instant conversational responses and systems must track tokens, response times and logs for each invocation. Models change every other week, so regression testing uses the Evals framework with model‑grade eval templates that run another LLM to semantically grade outputs. Key risks: biased training data, categorized legal obligations under GDPR/AI Act, frequent hallucination syndromes (e.g., answering ambiguous queries instead of asking), and exploitable flaws like prompt injection highlighted in LLM vulnerability lists. Observability and automated semantic checks are essential.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:00User Expectations
- 2:21Bias
- 4:13Legal Risks
- 5:11Hallucinations
- 6:35Regression Testing
- 8:56Nonfunctional Aspects
- 11:15Closing Thoughts
related talks
26mHow Bad Actors AbUse LLMs The Mechanisms That Enable Misuse and What We Can Do About
55mLessons Learned From Implementing Generative AI Solutions
19mTesting LLM-Powered Applications
01h 04mWhat are we thinking in the age of AI?
33mAI is coming for your job
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
40mKeynote
42mQA for ML: How we can trust AI with Food Sustainability