Extending Envoy: A Guide to Custom Extensions with Envoy Gateway
Talkat EnvoyCon NA 2024 · on Tuesday, 2024-11-12· at 11:15· Hyatt Regency | Level 2 | Salt Lake Ballroom C·
summary
Envoy Gateway loads custom extensions either as WASM or external gRPC processor and configures them via an Envoy ExtensionPolicy; demo uses Unix domain sockets, mTLS, health checks, circuit breakers and body buffering. The team builds an llm scheduler that scrapes each VM pod for loaded LoRA adapters and queue/cache metrics, returns a header with the chosen pod IP, and performs adapter-aware routing via header mutation. In the demo two LLaMA adapters (tweet summary and SQL) stay affinity routed, yielding higher throughput and lower latency under batch traffic.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:10Upgrade from Ingress to Envoy Gateway
- 2:36Custom policies and filters
- 4:41Extension policies and WASM support
- 7:56External processing filters
- 9:01Security and resilience
- 11:34Advanced deployment patterns
- 11:54WASM vs external processing
- 14:28Kubernetes LLM Gateway
- 16:35Laura-aware load balancing
- 19:42Demo setup
- 25:03Q&A
related talks
28mExtending Envoy for Our Use: An End User Adoption Story from Fidelity
32mHow Lyft Used Envoy to Rethink Microservice Development
14mKeeping Envoy Gateway On, Even When Bad Things Happen
26mBuilding a Dev/Test Loop for a Kubernetes Edge Gateway with Envoy Proxy
29mSafely deploying a 100K line Envoy YAML configuration to production
19mSafely Deploying a 100K line Envoy YAML Configuration to Production
33mDeveloping & Debugging WebAssembly Filters for Envoy
31mWhen LLMs Hit Production: Why You Need an AI Gateway