AI Package Hallucination: Spreading Malicious Packages Using Generative AI
summary
Claim: generative models like jgpt gini CH GPT and GPT 3.5 GPT 4 suggest package names that do not exist then attackers publish malicious packages with those names. Method: collect ~48,000 how‑to questions across five languages, query four models, extract hallucinated package names, upload payloads, verify repetitiveness and cross‑model recurrence. Results: ~30% hallucination rate overall, 65% hallucination rate in Gemini, repeated names across models, and a published package that records 377k downloads before removal.
AI-written summary · about
chapters (15)
- 0:00AI package hallucination intro
- 2:07Generative AI risks
- 3:55Over-reliance on AI
- 6:04AI hallucinations explained
- 9:00Why models hallucinate
- 10:27Supply chain attacks overview
- 13:14First research on AI packages
- 16:04Research process
- 19:37Proof of concept demo
- 23:13Motivation for second study
- 23:57Second research method
- 26:52Results across models
- 29:27Real-world attack examples
- 31:37Impact on repositories
- 33:25Safety recommendations
related talks
26mHow can we abuse AI hallucinations to feed the already bloated software supply chain
55mAI Showdown: The Good, The Glitchy, and The Risky
16mHarnessing AI for Malevolence: A Hackers Tale
32mSecurity Implications of Artificial Intelligence Synthesizers
41mThe Unreliable Computer Revolution - Embracing the Double-Edged Sword of GenAI
44mAI Foundations: Mitigate Risks and Boost SOC Efficiency
28mWhat Are AI Hallucinations?
31mHacking AI Applications