Is Your Service Mesh AI Ready?
Talkat Istio Day North America 2025 · on Monday, 2025-11-10· at 13:35· Building B | Level 4 | B406b-407·
summary
Istio already provides API key management and ingress/egress control but falls short on AI specifics: request/response bodies and streaming SSEs need logging to see prompts and completions, rate limits must be token-based not request-based, and observability needs time-to-first-token and per-token metrics. Inference gets real wins via the Gateway API inference extension and optimized load‑balancers for VLMs; MCP (Model Context Protocol) is <1 year old with rapid revisions and requires new stateful, streaming and auth support.
AI-written summary · about
chapters (11)
- 0:00AI and service mesh intro
- 1:43AI infrastructure needs
- 3:08LM consumption challenges
- 8:43LM solutions in mesh
- 11:57LM load balancing
- 15:48LM serving with mesh
- 16:11Model Context Protocol intro
- 20:22MCP challenges in mesh
- 22:31AI integration future
- 24:15Ambient mesh and AI gateways
- 26:18Summary and Q&A
related talks
26mAI in My API Gateway ... But Why?
26mAI Traffic Is Different: Rethinking Proxy Arc...
31mWhen LLMs Hit Production: Why You Need an AI Gateway
04mEnvoy Today: What’s New In Managing Cloud-Native And AI Workl...
30mAI APIs: Taming the Wild West with AI Gateways
45mRun, Secure, and Govern Your AI Traffic with an AI Gateway for Multi-LLM
38mAI Engineering Stack: Assisted to Delegated
31mAI APIs: Taming the Wild West with AI Gateways