When LLMs Hit Production: Why You Need an AI Gateway
summary
Envoy AI Gateway is a Kubernetes-native control plane that normalizes OpenAI‑style APIs and multiplexes traffic across OpenAI, Anthropic, Bedrock, Gemini and self‑hosted models while enforcing token-aware rate limiting and per‑tenant quotas (rate limits are on tokens not requests). It provides CRDs (AI Gateway Route, AI Service Backend, Backend Security Policy), cross‑provider failover like model X fallback to Anthropic, and intelligent self‑hosted routing via endpoint picker (EPP) tied to VLLM metrics. Project began Nov 2024, has four releases and is nearing 1.0.
AI-written summary · about
chapters (16)
- 0:00AI gateways for LLMs
- 1:26GenAI vs traditional APIs
- 3:14Scaling LLM usage problems
- 5:20What is an AI gateway
- 6:46Envoy AI Gateway choice
- 7:08Unified API and rate limiting
- 9:56Tracking LLM costs
- 10:39Model-specific rate limiting
- 12:04Cross-provider failover
- 13:47Envoy AI Gateway CRDs
- 16:17Data flow in AI gateway
- 17:42Routing for self-hosted models
- 20:50Managing MCP servers
- 24:02Putting it all together in an enterprise
- 24:47Adopters and status
- 25:51Q&A
related talks
15mTaming AI Sprawl: Your First Look at the Envoy AI Gateway
17mSeamless AI-as-a-Service across Kubernetes Clusters With Envoy AI Gate...
45mRun, Secure, and Govern Your AI Traffic with an AI Gateway for Multi-LLM
17mThe Future of AI Traffic: What's New in Envoy AI Tra...
26mAI in My API Gateway ... But Why?
37mEvolution of Envoy AI Gateway
30mAI APIs: Taming the Wild West with AI Gateways
04mEnvoy Today: What’s New In Managing Cloud-Native And AI Workl...