One guardrail is not enough: Defense-in-depth against LLM jailbreaks
summary
Production chatbots leak contracts and data; OWASP ranks prompt injection #1 and RAG/fine-tuning don’t remove it. The pipeline runs on AWS with a fine-tuned BERT classifier on SageMaker, moderation/PII filters, an LLM canonicalizer/rephraser, hostile‑input retrieval checks, guarded generation and output filtering; a demo automates an asynchronous LLM‑as‑judge triage that labels prod attacks for retraining. Cited PoisonRAG: injecting 5 malicious docs into 1M yields ~90% successful jailbreaks, so the layered defense continuously hardens cheaper filters.
AI-written summary · about
chapters (14)
- 0:00LLM guardrail challenges
- 1:06Introduction
- 2:08LLM failure incidents
- 4:58Industry security risks
- 5:59Two-tier jailbreak defense
- 8:07Executable guardrail policies
- 10:14Stacking defense layers
- 13:33Demo: Testing guardrails
- 17:16Balancing latency and cost in defense layers
- 18:44Feedback loops for improvement
- 20:32AutoResearch for guardrails
- 23:20Risks of automated tuning
- 24:22Key takeaways
- 25:26Q&A
related talks
28mJailbreaking and Securing LLM Apps: Lessons from an online wargame experiment
46mGuarding the LLM Galaxy: Security, Privacy, and Guardrails in the...
29mBeyond the Hype: Building Trustworthy and Reliable LLM Applications with Guardrails
43mOne Poisoned Artifact Can Steer Your AI: How Robust Are Your LLM-Assisted Security Workflows?
44mSecuring LLM-Powered Apps: Overcoming Security and Privacy Challenges
21mProtect your generative AI applications against jailbreaks
01h 01mIntroduction to AI Security
59mRed Teaming Large Language Models