Behind the Firewall: Designing Secure LLM Pipelines for Regulated Data
summary
Reference architecture shows running LLMs entirely inside the security perimeter with concrete artifacts: cryptographic model signing to verify provenance, a zero‑trust RAG pipeline that enforces real‑time authorization with dual checks at query time, and concrete sizing guidance (e.g., a 7B model on one A100 for ~25 RPS versus 70B on ~8 A100s). Code‑level levers include int8 quantization for cost cuts and cache repetitive summaries for huge throughput savings, plus eval harnesses auditors can ingest.
AI-written summary · about
chapters (10)
- 0:00Secure LLM pipelines intro
- 3:45API calls aren't safe enough
- 7:19Regulatory challenges in AI
- 9:27Threat modeling for LLM systems
- 12:14Production-first mindset
- 18:40Real-time auth in RAG
- 22:14Model provenance verification
- 24:24Pitfalls in production deployment
- 30:45Key lessons and best practices
- 32:58Q&A on costs
related talks
47mA Step-by-Step Guide to Securing Large Language Models (LLMs)
28mNavigating LLM deployment: Tips, tricks, and techniques
25mLLM SecOps
36mLLMOps in Practice- Building Secure, Governed Pipelines 4 LLMs
28mAnonymization in the LLM Era: No Room for Compromise
30mLLMs: Beyond the Hype - A Practical Journey to Scale
30mSecuring AI Workloads: Building Zero-Trust Architecture for LLM Applications
38mPhishing LLMs: Reeling in the Machine