Hype and Reality: How to Evaluate AI/ML in Cybersecurity
summary
Practical checklist for vendor claims: distinguishes generative models (ChatGPT, DALL‑E) from supervised, unsupervised and reinforcement methods; names products like QRadar Advisor with Watson and Microsoft Security Copilot and testing resources such as ML Commons/MLPerf and DataPerf; warns of concrete failures—data poisoning, Trojans in model zoos (Hugging Face), membership inference, reward‑hacking and model drift—and recommends measuring inference accuracy, verifying training data provenance, keeping a human in the loop and running vendor‑blind proofs in the buyer’s own environment.
AI-written summary · about
chapters (18)
- 0:00Introduction and level set
- 2:06History of AI hype
- 5:16AI vs ML differences
- 8:53Narrow vs general AI
- 10:18Generative AI explained
- 17:46Trusting AI outputs
- 18:06Supervised vs unsupervised ML
- 25:24Reinforcement learning risks
- 27:14AI risk management framework
- 33:19Evaluating AI performance
- 35:48Key questions for vendors
- 38:00Bias and model drift
- 40:15Human oversight necessity
- 42:20Threat modeling AI systems
- 43:04ROI and claims validation
- 45:12Solving real problems
- 47:21Q&A on pre-trained models
- 48:25Q&A on Zero Trust in AI
related talks
32mAI/Machine Learning: What is Actually Working in Cybersecurity?
49mThe Intersection of AI and Cybersecurity
01h 05mNavigating the Security and Privacy Landscape of Modern AI
44mAdvancing Cybersecurity with AI and Machine Learning: Opportunities and Challenges- 44mAdvancing Cybersecurity with AI and Machine Learning: Opportunities and Challenges
39mForward Focus: Perspectives on AI, Hype, and Security
07mCybersecurity Analysts' Perception of AI Security Tools and Practical Implications
24mHow to Build Trustworthy AI