Run, Secure, and Govern Your AI Traffic with an AI Gateway for Multi-LLM
summary
Kong builds an AI Gateway on top of the Kong Gateway (39K GitHub stars, ~350B requests/day) that fronts many LLMs behind one uniform API endpoint, centralizes secrets so the AI key is stored in the gateway with OIDC/authn, and enforces token-based rate limits (demo shows per-minute quotas and 429s). It adds semantic caching, routing, and prompt-guards using vector storage (Redis) to return cached answers with zero token cost, emits logs/metrics to ELK/Datadog/Prometheus, and integrates with OpenAI and LangChain SDKs.
AI-written summary · about
chapters (13)
- 0:00AI Gateway introduction
- 2:56Kong AI Gateway pillars
- 6:28Demo setup and proxy
- 10:29Token consumption controls
- 13:03Rate limiting and templates
- 18:55Legacy API with AI
- 21:03LLM observability and costs
- 25:04OpenAI with Kong Gateway
- 30:12Developer portal for AI
- 32:23Semantic AI overview
- 35:57Semantic caching and guarding
- 42:34Semantic routing for models
- 45:09Summary
related talks
31mWhen LLMs Hit Production: Why You Need an AI Gateway
41mFrom Chaos to Order: How to Konnect, secure and observe [...]
30mAI APIs: Taming the Wild West with AI Gateways
31mAI APIs: Taming the Wild West with AI Gateways
30mSecuring AI Workloads: Building Zero-Trust Architecture for LLM Applications
15mTaming AI Sprawl: Your First Look at the Envoy AI Gateway
17mSeamless AI-as-a-Service across Kubernetes Clusters With Envoy AI Gate...
26mAI Traffic Is Different: Rethinking Proxy Arc...