How to DDOS yourself with Envoy (and other tales of migration horror)
summary
Yelp migrates hundreds of services from SmartStack/HAProxy to Envoy by building a tiny Python control plane (under ~1,000 lines), rolling out via client-library feature toggles and tracer‑bullet pages, and converting Envoy gRPC access logs into HAProxy‑formatted syslog with a log‑transform shim. Deployment uses phased rollouts and docker-based end-to-end tests (docker‑compose) against master so control‑plane changes hit production confidently. A misconfigured health‑check update briefly DDOSes backend services, but phased deploys and fast rollback limit impact.
AI-written summary · about
chapters (15)
- 0:00Start
- 2:50Things that make developers
- 4:23Let's talk about Finagle
- 5:112014 - 2018)
- 12:31meshd - A Really Simple Control Plane
- 13:19Tracer Bullet
- 14:58Client Libraries
- 18:01Observability
- 19:28Cut Points
- 19:51The Old Way - Manual Testing and Luck
- 20:28The New Way - Automated E2E Tests
- 21:31Remember the Test Pyramid!
- 21:44Continuous integration, deployment
- 22:20Things will break at scale
- 23:33Summary
related talks
26mService Mesh Failure Stories at Scale
29mThe rise of Layer 7, microservices, and the proxy war...
22mThe Trials and Success of Adopting Envoy at Tinder
43mAirbnb Service Discovery: Past, Present, Future (Challenges of Change)
09mLightning Talk: The Hardest Part of Operating a Service Mesh: Envoy Proxy
22mDEVOPS DAYS MTL 2018 - Alexandre Gervais
25mAdapting Our XDS Control Plane for Proxyless gRPC
31mThe Anatomy of a Cascading Failure