From the Trenches: Automated Testing of Large Language Models
summary
Small startup builds an “AI insights” feature that uses LLMs to surface fraud signals, and implements automated testing that focuses on concrete properties not exact text — checks for specific terms, response length ranges, and absence of banned content. Tests run in CI with synthetic demo workspaces plus vetted production columns (clients gave permission), use Python + pytest + LangChain and an auto-evaluator tests pattern where an LLM answers yes/no against criteria. A surprise: ~600 manual responses caught a politically sensitive false positive; prompt refinements and tests eliminate it, costs stay around $50/month.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:31LLMs in our product
- 3:18Hackathon for LLM use cases
- 5:04Fraud detection with AI
- 7:36Data privacy and model refinement
- 9:03LLM testing challenges
- 10:30Testing non-deterministic models
- 12:16Synthetic test data for fraud
- 13:40LLM test automation principles
- 14:46Auto-evaluator tests
- 16:12Language for test automation
- 17:40Future of AI insights
- 18:45Adopt AI in testing
- 19:27Q&A: AI replace jobs?
- 20:09Q&A: AI tools for testers
- 22:00Closing
related talks
14mImproving Large Language Model Performance
49mSecuring AI Apps with the OWASP Top Ten for Large Language Models
19mTesting LLM-Powered Applications
24mResponsible AI: Assessing the Truthfulness of AI Model Responses
54mAI in Testing: Hip or Hype?
32m🧐 A Look at Possibilities of Using AI in #SoftwareTesting
13mNew Challenges in Software Quality Brought by GenAI and LLMs
46mHow Large Language Models Are Reshaping the Cybersecurity Landscape