Measuring the Difference: Metric Development at NCCoE’s Securing AI Testbed
summary
Describes using the NCCoE Securing AI Testbed to measure how well AI defenses perform against specific attacks, showing concrete measurement techniques applied to AI-enabled systems. Focuses on creating and applying metrics to evaluate resilience of AI technologies in hypothetical industry adoption scenarios.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 3:03A Taxonomy and Terminology of Adversarial Attacks and Defenses on Machine Learning (Draft NISTIR 8269)
- 8:23Deployment context also matters!
- 9:13Scenario testing, Parameter Sweeping, Evaluation
- 10:342nd Party Testing
- 15:07Supermarket example: Attack Profile
- 18:38Fruit Classification Combinations Tested
- 19:58Model Comparison-Clean Data
- 21:25Model Comparison-Adversarial Data
- 22:01Vendor 2 Details
- 24:43Key Takeaways of Supermarket Scenario
- 26:28Integration and Regression Testing
- 28:35Road Sign Detection Combinations Explored
- 29:11Dataset Considerations
- 30:32Performance Metrics
- 34:13Backdoor Poisoning Attack
- 36:04Key Takeaways from Road Sign Detection
- 38:59Apply what you have learned
related talks
40mMaking Sense of the Landscape of Attacks and Defenses Against AI
18mAutomated Risk Evaluation and Mitigation for AI Systems
28mAI Under the Hood: Unmasking Hidden Threats
44mAI Meets Cybersecurity: Boosting Speed Without Losing Control
55mBenchmarking Threat Resilience Metrics aka How Do We Compare to Our Peers?
18mCommit Secure AI: Embedding Security in Code, Pipelines, and Production
45mAI Security, from PoC to Production
49mA National Initiative for Cybersecurity Advancement