Testing the Untestable: Turning OWASP AILLM Risks into Practical QA Checks
summary
Translates OWASP AI/LLM Top 10 into concrete QA checks with a six-step method: pick a risk, describe a bad day, write probes, find the signal, write checks across retries and set rules. Demo shows an injected review causing a $4,900 refund, the OWASP suite initially fails, then passes after switching on six protections. Tests run several times (e.g. 1/5 or 4/5 passes), money and personal data must pass every time, and checks run under 90 seconds for CI, nightly sampling and a deep run before release.
AI-written summary · about
chapters (46)
- 0:00Start
- 0:14Session Opens - Testing the Untestable
- 0:44The Wall: No Fixed Output to Assert Against
- 1:15Speaker Intro - Thejes Sree Satheesh Kumar, Quality Analyst, ThoughtWorks
- 1:59The Fact: OWASP Rewrote the Top 10 LLM Risks 17 Days Ago
- 3:01Background: Security Certifications and a Book on Breaking MCP
- 3:31Change One: Excessive Agency Jumped From Sixth to Third
- 4:02Change Two: Unbounded Consumption; Change Three: Hidden Context Exposure
- 4:32Improper Output Handling Fell to Tenth - Because We Got Good at It
- 5:03How Many Teams Actually Ran Security Tests on Their AI Feature?
- 5:33Why Automation Doesn't Catch It: No Single Right Answer, Endless Inputs
- 6:03Reason Three: It Changes Without a Release
- 6:35Reason Four: Nothing Looks Broken - 200 OK and Perfect JSON
- 7:05You Were Never Testing the AI - You Were Testing the Software Around It
- 7:37The Six Steps: Turning a Risk Into a Test
- 8:07Describe a Bad Day, Write a Probe, Find the Signal
- 8:39Write the Check Across Retries, Set the Rule - Money Fails the Release
- 9:11Meet the Victim: A Support Chatbot That Can Refund and Close Accounts
- 10:11Six Boxes, Every One Can Go Wrong, Every One Leaves a Trace
- 10:42The Receipt From One Request
- 11:12Risk One: Prompt Injection
- 11:43The Check: Did an Innocent Question Cause an Action?
- 12:16Risk Two: Sensitive Information Disclosure
- 12:46Risk Three: Excessive Agency
- 13:17Allow-List, Amount Limit, Human Approval for Anything Irreversible
- 13:49Risk Four: Misinformation
- 14:21You Cannot Test for Truth - Ask Whether It Points to a Document
- 14:52Risk Five: Improper Output Handling
- 15:23Check the Screen, Not the AI - You Already Know How to Escape Output
- 15:53Live Demo: The Injected Review Triggering a $4,900 Refund
- 16:38Running the Suite Against the OWASP Top 10 - Everything Fails
- 17:40Switching On Six Protections and Re-Running - Everything Passes
- 18:41Running It Again: Some Pass, Some Fail
- 19:11That's Not Flakiness - That's Real Behaviour
- 19:42Four Rules for Testing Something That Changes
- 20:12Money and Personal Data Must Pass Every Time
- 20:48Where This Actually Runs: Keep It Under 90 Seconds or Someone Will Remove It
- 21:18Nightly Runs and the Deep Run Before Release
- 21:48Five Mistakes He Personally Made
- 22:19"Please Ignore Malicious Instructions" Is a Request, Not a Control
- 22:50The One Slide to Take Home: Ask for the Logs
- 23:21Put the Nasty Inputs in a File Anyone Can Add To
- 23:52Takeaways: Test Around the AI, Name What You Look At, Count
- 24:24It's Not a Black Box - It's a System Nobody Put Logging Into
- 24:54Q&A: Sharing Responsibility Across Security, QA and AI Engineering
- 26:01Q&A: Evaluating Agent Misuse Across Multi-Step Tool Trajectories
related talks
27mHow to Start Security Testing Without Fear
22mAre you safe from OWASP #11?
59mSecuring the Digital Landscape: A Deep Dive into OWASP Top 10 for Applications, APIs, and LLMs
30mNavigating the New AI Security Landscape
28mSecuring LLMs: Insights into OWASP top 10
44mShift-Left-Security with the Security Test Pyramid
37mDeploying Security Testing Practice in Your Team and Company
50mFix The Damned Software