Service Mesh Failure Stories at Scale
summary
Allegro.pl migrates roughly 800 microservices across ~5,000 instances using Envoy plus a custom Java/Kotlin control plane (open sourced) to support Mesos, VMs and a Consul discovery backend; deployment triggers out‑of‑memory due to Envoy memory reporting gaps, default 1024 per‑worker connection limits, and timeouts/headers behavior that break long requests and case‑sensitive sanitizers; fixes include ordering EDS/RDS updates, a hot‑restart bug patch, connection reduction heuristics and resilience tests that drive stability at 1 million RPS and prove transparent proxies are not transparent while using java control plane base for upgrades.
AI-written summary · about
chapters (11)
- 0:00Introduction to failure stories
- 2:33Why build our own mesh
- 9:12First deployment problems
- 11:45Proxy behavior changes
- 14:16Timeout and connection limits
- 16:47HTTP protocol strictness
- 17:49Control plane challenges
- 21:01Performance optimizations
- 22:30Resilience testing
- 23:13Lessons learned
- 24:37Future plans and thanks
related talks
56mService Meshes - Powering the next wave of microservice architectures
39mGrow with Less Pains - Meshing From Monolith to Microservices
45mService Mesh War Stories - Lessons Learned the Hard Way
36mService Messy: Pagerduty's Service Mesh Migration
53mService Mesh: What, Why, And How?
25mHow to DDOS yourself with Envoy (and other tales of migration horror)
28mEnvoy at Square
22mThe Trials and Success of Adopting Envoy at Tinder