Securing LLMs in Kubernetes: Best Practices
summary
Deploys an open LLM (uploader lists Mistral) on Kubernetes and front-ends it with NGINX + Istio-style gateways, using an Envoy Lua filter that forwards every prompt and response to a dedicated prompt-monitor service which returns 200/400 to allow or block requests. Detection uses token and contextual metrics (BLEU/BERT-style and embedding cosine similarity), plus PII/entity recognition and toxicity models; demo shows blocking prompt injections and jailbreaks and returning sanitized responses via egress gateways.
AI-written summary · about
chapters (7)
- 0:00GenAI security importance
- 3:11Unique challenges of large language models
- 6:21Prevention and mitigation
- 8:31K8s setup for LLM security
- 10:38Prompt injection detection demo
- 15:20Hallucination detection
- 24:34Summary & Q&A
related talks
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
30mSecuring AI Workloads: Building Zero-Trust Architecture for LLM Applications
25mLLM Powered Agents with Kubernetes
27mSecuring The Gateway And Mitigating Risks In LLM API Integration
33mllm-d: Kubernetes Native Distributed Inferencing
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...