Surfacing Hidden Privacy Risks in Code: Lessons from LLM and Retrieval Assisted Detection
summary
A detection pipeline evaluates 183,000+ methods across nine repositories using GPT-4 with either fixed few-shot prompts or a retrieval-augmented generation (RAG) layer backed by 84 curated incident examples in Azure AI Search. RAG finds twice as many confirmed Default Grant Access issues as the static prompts (RAG doubled confirmed findings), but overall precision does not improve and drops in TypeScript and C++ (false positives remain high), forcing a tradeoff between recall, cost, and developer trust.
AI-written summary · about
chapters (7)
- 0:00Hidden privacy risks detection
- 1:14Mundane privacy mistakes
- 3:00Default grant access vulnerabilities
- 6:32LLM vs RAG detection quality
- 10:01Analysis of 183k methods
- 14:59Cost comparison and lessons
- 16:44Q&A
related talks
30mNavigating the New AI Security Landscape
51mFrom Text to Flaws: vulnerabilities in applications with Generative AI and LLMs
41mMerging Human Cyber Security Expertise with Generative AI
45mSecuring AI: There Is No Try, Only Do!
35mAugmented Retrieval Products with GAI models
29mOne-Click Code Fix: Securing Code Using AI
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
26mOne guardrail is not enough: Defense-in-depth against LLM jailbreaks