How Lyft Used Envoy to Rethink Microservice Development
summary
Lyft replaces slow per-engineer “one‑box” EC2s with a shared staging overrides system that lets developers run isolated service versions in staging or on laptops. It uses Envoy custom HTTP filter logic, an xDS/EDS control plane, trace baggage headers carrying JSON override metadata, an original destination cluster driven by the X-Envoy-Original-DST-Host header, offloaded pods via a GitHub /offload flow and a scriptable ingress proxy. Provisioning drops from ~1 hour to ~10 minutes, parity with prod improves, and isolation generally works; challenges remain with header propagation, alerting noise and infra testing.
AI-written summary · about
chapters (17)
- 0:00Start
- 0:13Developer Experience
- 0:56The change pipeline
- 4:32Advantages of staging
- 8:06Introducing: Staging Overrides
- 9:01Offloaded pods: EDS exclusion
- 10:20With override metadata
- 11:20Distributed tracing
- 12:10OpenTracing (OpenTelemetry)
- 14:10Filter logic
- 14:51Implementation
- 17:09Set Original Destination Cluster
- 17:31Summary
- 18:56Baggage-attachment tooling
- 21:21Custom typescript proxy
- 23:28Long-term vision for staging overrides
- 25:14Challenges / retro
related talks
35mThe Mechanics of Deploying Envoy at Lyft
29mSafely deploying a 100K line Envoy YAML configuration to production
19mSafely Deploying a 100K line Envoy YAML Configuration to Production
14mChanging Lanes with Lyft
30mExtending Envoy: A Guide to Custom Extensions with Envoy Gateway
37mBottoms-Up Adoption of a Microservices Workflow Using Kubernetes & Envoy
06mLightning Talk: How a Global Logistics Company Migrated Clouds with Zero Downtime
34mRunning Large-Scale Stateful Workloads On Kubernetes at Lyft