How continuous testing keeps your LLM on track
summary
Argues that continuous testing catches regressions and drift by combining an evolving golden dataset, runtime logging, and scheduled checks; shows practical tools like Evidently for reports and Hugging Face classifiers for toxicity/PII, plus sandboxed execution to verify generated code actually runs. Uses semantic similarity and simple proxy metrics or a LLM as a judge to detect hallucinations and behavior changes, and reports that these layered tests surface real failures and guide prompt updates.
AI-written summary · about
chapters (6)
- 0:00Continuous testing for LLMs
- 3:53Monitoring limits for LLMs
- 13:44LLM evaluation scenarios
- 27:36Proxy metrics for LLMs
- 31:29LLM evaluation tips
- 34:44Q&A: Datasets and reliability
related talks
38mMaking LLMs reliable - A practical framework
35mEvaluating LLM systems: the nuts and bolts
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
15mLLMs and Test Automation: A New Era of Self Healing
25mEvaluating An LLM's Ability To Code
29mCustomizing and Evaluating LLMs, an Ops Perspective
28mTrustworthy LLMs: Vibe checks are not all you need
33mExplore LLMs and Multimodal Approaches in Test Automation