Evolve, Adapt, and Adopt: Envoy in the AI Era
summary
Panel describes concrete changes: Envoy teams add an Envoy AI Gateway layer, an inference pool model and hooks (external processing server, Lua, WebAssembly) to make routing decisions from live GPU/LLM metrics; reports that initial load‑balancing design is too slow so Google rebuilds it into an external load balancer and implements cross‑service failover and memory offload proposals for huge multimodel requests; adopters use token‑based routing and have already shipped MCP routing in AI Gateway and a Tetrate agent router service that proves the approach at scale.
AI-written summary · about
chapters (4)
- 0:00Panelist introductions
- 4:24Enabling AI traffic
- 13:56Real-world adopter stories
- 23:32Lessons and future outlook
related talks
17mThe Future of AI Traffic: What's New in Envoy AI Tra...
27mEnvoy Proxy: Evolved for Serving LLMs
30mBeyond Monolithic AI: Cloud Native Patterns for Dynami...
12mInside ENVOY: Siemens' Production-Grade Agentic AI Services on
32mExpl(AI)n Like I’m 5: An Introduction To AI-Native Networking
14mReliable Agentic AI, the Open Way
45mUnifying AI experience for customers: Building multi-agent systems with Google ADK
29mSpanning the Globe with Envoy at Stripe