~/talk

Single-Layer LLM Guardrails Fail: 390 Prompt Injection Tests

Talkat Conf42 Prompt Engineering 2026 · on Thursday, 2026-10-29· [ai-assisted-development, security]
links:none on file
slides:none on file

A 390‑prompt evaluation (172 single‑turn attacks, 18 multi‑turn attacks, 200 legitimate prompts) shows a single-layer guardrail with keyword rules catches 22.6% of attacks (F1 0.368), catching 43 of 190 attacks, missing 147 and producing 1 false alarm, while a semantic intent classifier that reads up to five prior turns detects 96.8% (F1 0.984), catching 184, missing 6, and producing zero false alarms. Three structural failures of one‑message filters motivate a five-layer architecture (fast <100 ms layer one filter, layer two semantic check, audit log, DynamoDB, tool authorization, output checks, human gate). An evaluation error from API throttling is noted and corrected.

AI-written summary · about

  1. 0:00Introduction and the Single-Guardrail Problem
  2. 1:09Why One Filter Misses Attacks
  3. 2:27The 390-Prompt Test Set
  4. 3:58Four Detection Approaches Compared
  5. 6:05Results: Keyword vs Semantic Classifier
  6. 7:52Three Structural Limits of Single Layers
  7. 9:04Dual Detection Architecture
  8. 10:36Inside the Layer Two Classifier
  9. 12:10Four Example Requests
  10. 14:02Raw Counts and False Alarms
  11. 15:39My Evaluation Mistake
  12. 18:42Five-Layer Architecture for Agents
  13. 20:19Key Takeaways and Resources
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.