Lightning Talk: Serving Guardrail Detectors on Vllm
summary
An open-sourced vLLM detector adapter adds detector endpoints to a vLLM server so Granite Guardian, Llama Guard or simple regex/encoder detectors run through the same host and API, maintaining vLLM benefits like continuous batching and no added network hop. The adapter maps model outputs (for example Llama Guard S1/S2 labels) into human categories, returns classes and scores, and shows cases like a high jailbreak risk with high confidence. Code is on GitHub under Apache-2 and is ready to plug into existing completions/chat endpoints.
AI-written summary · about
related talks
11mNew Protection Tools for Llama Developers
33mLlamaFirewall: An open source guardrail system for building
29mBeyond the Hype: Building Trustworthy and Reliable LLM Applications with Guardrails
37mSecure by Design: Strategies for LLM Adoption in Cloud-Native Environm...
38mMaking LLMs reliable - A practical framework
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
01h 49mAI engineering with open access LLMs that lie, curse, and steal
10mWIP: From Detection to Explanation: Using LLMs for Adversarial Scenario Analysis...