~/talk

Evolution of Envoy AI Gateway

on YouTube 37m
Talkat EnvoyCon North America 2025 · on Monday, 2025-11-10· at 09:45· Building B | Level 4 | B406b-407· [ai-assisted-development, networking]

speakers

name:Yan Avlasov
links:none on file
name:Takeshi Yoneda
website:https://mathetake.github.io/

resources

video:https://youtu.be/oXlnhRKPXhA
slides:none on file

summary

Envoy AI Gateway uses Envoy Gateway + xDS to serve GenAI on Kubernetes and now implements token-based global rate limiting by extracting token usage from responses into dynamic metadata and decrementing quotas on stream close; it does transparent cross‑provider failover by running conversion, auth, and signing logic per retry in an xrog upstream filter; it supports self‑hosted pools with an external endpoint picker that scrapes Prometheus for queue depth and picks hosts, shows measurable LB improvements, and notes limits (output tokens unknown until response, work in progress).

AI-written summary · about

chapters (12)

  1. 0:00Envoy AI Gateway intro
  2. 1:23AI LLM traffic challenges
  3. 3:50Project history & milestones
  4. 4:56High-level architecture
  5. 5:57Token-based rate limiting
  6. 9:30Cross-provider failover
  7. 12:44Self-hosted model support
  8. 17:18Intelligent load balancing
  9. 21:55Benchmarking results
  10. 22:17Project state & roadmap
  11. 26:11Envoy in AI proxy landscape
  12. 28:44Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.