Your Test Suite Can’t Catch a Hallucination: Talk on AI in Production
summary
A concrete failure mode: an agent told a borrower their loan was declined for low credit instead of missing proof of income, because outputs blend database facts with generic KB reasons. The talk advocates building a golden standard dataset, running automated CI/CD evals with binary pass/fail plus critiques, using an LLM-as-judge and human-in-loop error analysis, instrumenting tracing with Langfuse/Arise, and designing an internal viewer that cuts iteration time roughly tenfold to approach 99.999% reliability.
AI-written summary · about
chapters (9)
- 0:00AI hallucinations in production
- 3:56Why testing fails for AI
- 11:48Tests to rigorous AI evaluations
- 16:31Continuous evaluation framework
- 22:59Pass-fail and error analysis
- 29:00Advanced AI agent evaluations
- 32:13Tools and infrastructure for AI QA
- 37:36AI reliability and trust mindset
- 46:14Q&A
related talks
30mWhen AI Takes Over: DevOps Nightmare Unleashed
37mAI Hallucination Explained 🤯 Why AI Gets Things Wrong
42mRobust Testing Strategies for AI-Driven Applications
28mWhat Are AI Hallucinations?
01h 04mWhat are we thinking in the age of AI?
23mBeyond Tests What to Verify in AI Generated Code
43mTesting Agentic AI Applications: Beyond Traditional QA
26mReducing Hallucinations and Evaluating LLMs for Production