ML-Driven Test Intelligence at Scale - What Works What Fails and Why It Matters
summary
Claims a production-ready dual‑layer ML pipeline: XGBoost and scikit-learn for test selection plus an LSTM autoencoder (TensorFlow/PyTorch) for anomaly signals, implemented as the open source Log Miner QA. Shows redaction-first privacy, isolation forest and embedding distance for journey anomalies, and a prioritisation metric: Novelty, Exposure, Impact, Reproducibility. Demonstrates that LLM‑generated tests often prove nothing, that 95% aggregate agreement can hide dangerous 5% slices, and prescribes a sanitize→replay→define invariant→promote closed loop.
AI-written summary · about
chapters (48)
- 0:00Start
- 0:07Session Opens - Beyond the Hype: ML-Driven Test Intelligence at Scale
- 0:38The Promise: Faster Feedback, Smarter Selection, Anomaly Detection
- 1:09The Setting: One of the Largest US Financial Institutions
- 1:40Speaker Intro - Tanvi Mittal, Software Quality Lead, US Bancorp
- 3:19Generating Tests Has Become Cheap - That's Not the Hard Problem
- 3:50The Real Question: What Should We Test Next?
- 4:21The Experiment: Copilot Writing a Parameterised Query Correctly
- 4:53Not a Knowledge Problem - an Activation Problem
- 5:25How Stable Is a Known Capability When Context Changes?
- 5:56Four Assumptions AI Breaks - and Why "Non-Determinism" Is Too Shallow
- 6:27A Bigger Change Surface: Model, Prompt, Retrieval, Tool Calls
- 6:57From Verifying Output to Measuring Behaviour
- 7:28Test Intelligence as a Layer Asking Three Continuous Questions
- 7:59The Model Writing the Test Is the Easiest Model in the Pipeline
- 8:30Four Ways Scale Actually Shows Up: Variance, Model Drift, Privacy, Auditability
- 9:00The Boring but Vital Question: Why Is This Test a Release Gate?
- 9:32The Closed Loop and Its Two Entry Points
- 10:02Observe, Prioritise, Reconstruct, Assert, Validate, Decide, Learn
- 10:34If a Production Incident Teaches You Nothing, the Loop Isn't Closed
- 11:06Should That Observed Journey Become a Permanent Evaluation?
- 11:39The Prioritisation Model: Novelty, Exposure, Impact, Reproducibility
- 12:09"High Risk Because GPT Said So" Is a Number, Not a Risk Model
- 12:41Why a Layered Evaluation Strategy Is Necessary
- 13:12Three Layers: Manual Review, CI/CD Evals, Production Monitoring
- 13:42Introducing Log Miner QA - One Implementation of the Loop
- 14:12Production Showed Behaviour the Test Suite Didn't Know About
- 14:43The Real Starting Point: How Do We Stop Learning the Same Lesson Twice?
- 15:46Privacy First - Redact Before the Intelligence Layer Touches It
- 16:18Regex, Domain Validators and spaCy NER With Stable Tokens
- 16:49Trap Two: Anomaly Detection Is Not Defect Detection
- 17:20A Rare Event May Be a Defect, a VIP, or Month-End Processing
- 17:52Isolation Forest and Embedding Distance for Journey-Level Anomalies
- 18:23Failure, Accepted Recovery Path, or Legitimate Rare Workflow?
- 18:53LLMs Produce Convincing Tests That Prove Almost Nothing
- 19:24The Oracle Problem: What Invariant Should Have Held?
- 19:54A Weak Test vs a Meaningful Invariant: The Security Control Must Trigger and Be Audited
- 20:28LLM as a Judge - a Word That Grants Unearned Authority
- 20:58Measure Disagreement by Slice, Not by Average
- 21:2895% Agreement Is Terrible If the 5% Is All Safety Cases
- 21:58Three Assumptions Carried Over From Traditional Automation
- 22:30Anomaly Does Not Equal Defect; Generated Tests Do Not Equal Coverage
- 23:00What to Actually Do Next Week - Start With One Flow
- 23:32Close One Loop: Sanitise, Replay, Define the Invariant, Promote
- 24:02Don't Install Four Eval Frameworks - Learn One Thing From Production
- 24:34Observe, Prioritise, Validate, Learn, Feed Back
- 25:06Q&A: Preventing Redundant Gherkin Tests as Log Volume Scales
- 26:42Q&A: Which Governance Model Works Best for Agentic QA
related talks
10mML-based Log Analysis for Faster Debugging
53mUnleashing the Black Box: Testing Machine Learning Systems
29mEnhancing Performance Test Results with ML & LLMs: Communication Beyond the Numbers
32mLightning Talks
46mSW Testing: Can ML save us?
30mHow to Properly Test ML Models & Data
22mFrom the Trenches: Automated Testing of Large Language Models
46mEnhancing Agent Reliability Through Evaluation-Driven Development