Seamless AI-as-a-Service across Kubernetes Clusters With Envoy AI Gate...
summary
A central GPU-enabled AI hub runs Envoy AI Gateway and a kube-bind backend and exposes model backends (OpenAI, Anthropic, Bedrock or self‑hosted) as Kubernetes API resources. Consumer clusters use kubectl to create a kube-bind APIServiceExport which syncs CRDs and optional ConfigMaps to the hub while secrets remain on the hub, and Envoy handles routing, token rate limits and telemetry. The pattern requires routable cross‑cluster networking and one extra critical cluster but makes onboarding a consumer a single kubectl bind command.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:23Multi-cluster LLM access challenges
- 2:05Central AI platform goals
- 3:09Envoy AI Gateway features
- 3:50KubeBind service binding
- 5:54AI hub architecture design
- 8:01Consumer cluster implementation
- 9:25Trade-offs and operations
- 10:49Try open source projects
- 11:31Q&A
related talks
15mTaming AI Sprawl: Your First Look at the Envoy AI Gateway
31mWhen LLMs Hit Production: Why You Need an AI Gateway
22mUnlock Kubernetes & Envoy: Master Your API Ecosystem!
04mEnvoy Today: What’s New In Managing Cloud-Native And AI Workl...
37mEvolution of Envoy AI Gateway
32mK8s-based API Gateway
17mThe Future of AI Traffic: What's New in Envoy AI Tra...
09mSponsored Keynote: Hand Meet Glove: Why Kubernetes Will Become the Platform of Choice for Agentic Architectures