AI Just Happened to Me : Help!!
summary
Covers concrete QA actions: prefers RAG pipelines over repeated fine‑tuning and retrains models only once, runs AB tests to compare old and new models, and builds a small JavaScript dashboard to plot eval results over time. Calls out real failures (an agent that deleted a production DB, a bot that ordered 250 chicken nuggets), pitfalls like overfitting from 500 cat photos, prompt‑injection and hallucination, and uses tools such as Microsoft Eval, OpenAI eval, Datadog and red‑teaming agents.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 4:43Talk agenda and testing basics
- 5:46What is an LLM and testing challenges
- 7:12Agentic AI and orchestration
- 9:23Fine-tuning and retraining models
- 11:28Agentic AI testing focus
- 16:00Testing RAG pipelines
- 17:25Managing RAG noise and hallucinations
- 18:51RAG reliability and latency
- 19:34Fine-tuning risks and biases
- 21:59A/B testing fine-tuned models
- 23:28Prompt engineering and context
- 24:31Security risks in prompts
- 26:17Tools for AI model evaluation
- 27:38Visualizing model performance
- 31:29AI in regulated industries
- 32:53Getting started with AI evaluation
- 33:57Gaps in AI testing tools
- 34:17Beginner advice for LLM testing
- 36:44Keeping up with AI advancements
- 38:07Q&A
related talks
43mAI for Testing – Panel Discussion
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
24mHow to Test AI Features Before Your Users Do
33mEvaluating Your AI Agent
57mEvaluating AI Agents in production with Python
15mPractical tactics to build reliable AI apps
54mAI: Actual Intelligence
55mAI-First Testing