~/talk

When Software Starts Thinking: The Now of Quality Engineering

on YouTube 41m
Talkat TestMu Conference 2026 · 2026-08-19..08-21· [ai-assisted-development]

speaker

name:Chinmaya Kothari
links:none on file

resources

video:https://youtu.be/5o42Orp1_78
slides:none on file

summary

Generative AI breaks the “same input, same output” rule, so quality work shifts into measurable AI Assurance: map each trust attribute to risks, pick metrics, run tests and record evidence (RMTE). The talk gives concrete numbers — 170+ metrics catalogued, a sample threshold of ≤5% hallucination and a composite release gate of >95% trust score — and shows RAG checks (citation sufficiency and citation accuracy), agent tool‑selection validation, and an audit trail that packages test results as evidence for production decisions.

AI-written summary · about

chapters (50)

  1. 0:00Start
  2. 0:04Session Opens - When Software Starts Thinking
  3. 0:35Generative AI Broke the Rule: Same Input, Same Output
  4. 1:06Speaker Intro - Chinmaya Kothari, Director of QE and AI COE, QualiZeal
  5. 2:08The Opening Example: "Will This Contract Clause Hold Up?"
  6. 2:38Same Question, Same System, Opposite Answers
  7. 3:08The Deterministic World: Printing the Same Paragraph Every Time
  8. 4:12How We Tested in That World - Any Change Was a Defect
  9. 4:43The Translator Example: Same Paragraph, Three Different Translations
  10. 5:44Interpretation Enters, and the Whole Testing Paradigm Changes
  11. 6:15The Generative AI Example: "Delivery Within 30 Days"
  12. 6:46Words Added, Meaning Changed - That's Hallucination
  13. 7:18Confidence Does Not Mean Correctness
  14. 7:48The Agentic Example: "Ship This Assignment by Friday"
  15. 8:19Message Drift as a Distinct Failure Mode
  16. 8:50Bias, Inaccuracy, Safety, PII and PHI Exposure
  17. 9:21The Tester's Role Changes: From Correct to Trustable
  18. 9:51The Paradigm Shift From Pass/Fail to Overall Trust
  19. 10:21The Trust Attributes You Have to Evaluate
  20. 10:51The NIST AI Risk Management Framework and Its Six Attributes
  21. 11:21If Even One Attribute Fails, the System Is Not Trustable
  22. 11:53The Four-Step Framework: Risk, Metric, Test, Evidence
  23. 12:23Step One - Risk: Where Can Each Trust Attribute Go Wrong?
  24. 12:54Citation Sufficiency as a RAG-Specific Reliability Risk
  25. 13:24Step Two - Metric: How Do You Measure Hallucination?
  26. 13:54170-Plus Metrics Defined So Far, and Still Growing
  27. 14:55AI Systems Hallucinate by Design - You Cannot Stop It
  28. 15:26Which Brings You to Thresholds
  29. 16:28Five Hallucinations in 100 Is Acceptable; Six Is Not
  30. 17:30One Risk, One Metric - Now Multiply It Across Everything
  31. 18:00Step Four - Test: Actually Measuring the Metric
  32. 19:00Packaging Everything as Evidence and an Audit Trail
  33. 19:30The Trap: One Passing Trust Attribute Is Not a Passing System
  34. 20:31Is 100% Trustable a Fair Requirement?
  35. 21:32A Composite Trust Score Above 95% as the Release Gate
  36. 22:03What Changed: We Used to Do Proofreading, Now We Fact-Check
  37. 23:37Why Citation Becomes the Key RAG Metric
  38. 24:39Citation Sufficiency and Citation Accuracy
  39. 25:40Recapping RMTE: Risk, Metric, Test, Evidence
  40. 26:43Back to the Contract Clause - That Wasn't a Bug
  41. 27:13An LLM Hardly Ever Says "I Don't Know"
  42. 28:17Too Much Creativity and Translation Becomes Fiction
  43. 30:37Q&A: Validating an Agent's Tool Selection Before Production
  44. 32:08Q&A: How Much Human Intervention Before an AI Test Can Be Trusted
  45. 33:41Internal Chatbot vs Customer-Facing - Very Different Bars
  46. 34:13Q&A: What Will Have Changed Most by Next Year?
  47. 35:15Q&A: Testing an Agent's Decision Process, Not Just Its Answer
  48. 36:47Only When Every Layer Is Trustable Is the System Trustable
  49. 37:17Q&A: Are Confidence and Correctness Together the Best Combination?
  50. 38:18Why Their Framework Lets You Weight Each Trust Attribute

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.