~/talk

The Future of AI Traffic: What's New in Envoy AI Tra...

on YouTube 17m
Talkat EnvoyCon Europe 2026 · on Wednesday, 2026-03-25· at 12:26· Hall 1-5 | Tram Zone | Cloud Native Theater· [large-language-models, ai-assisted-development]

speaker

name:Xiaolin Lin
website:https://xiaojl.dev

resources

video:https://youtu.be/7OkSrMBirmc
slides:none on file

summary

Envoy AI Gateway reaches production readiness in 2025 and now adds concrete 2026 features: a unified OpenAI-format API that dynamically translates to Azure AWS GCP Anthropic provider formats, token-based rate limiting across input/output, quota-aware routing that prefers provisioned capacity with policy-driven YAML rules, caching and batch file APIs for up to ~50% cost savings, and observability with per-model token metrics; measured performance shows millisecond data plane latency and control-plane scale to 2,000 routes with zero routing failures.

AI-written summary · about

chapters (12)

  1. 0:00AI traffic challenges
  2. 2:09Envoy AI Gateway project
  3. 3:14Project timeline and features
  4. 4:38Cost optimization 2025
  5. 8:25AI Gateway API stability
  6. 8:45Quota-aware routing
  7. 11:13Prompt caching strategies
  8. 12:42Batch processing for AI
  9. 13:46Response API for agents
  10. 15:11Multimodal AI overview
  11. 15:55Performance metrics and scaling
  12. 16:16Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.