Unraveling Istio: A Tale of Scale, Challenges and Triumphs at DevRev
summary
DevRev replaces a central gateway with edge gateways per service (Envoy sidecars) to remove a single point of failure, enable gRPC load balancing and circuit breaking, and move JSON-to-gRPC transcoding into sidecars using Bazel CI to generate and mount proto stubs. Upgrades use a canary control-plane roll with an automated data‑plane restart script; 502/503s drop by increasing Envoy drain to 300s, 504s from ALB require keep‑alive via an Envoy filter, and high Envoy CPU/memory stops after disabling distribution_tracking. Monitoring scales with federated Prometheus + Thanos and proxy_merge to cut scrape targets; a custom operator for smoother upgrades is in progress.
AI-written summary · about
chapters (21)
- 0:00Istio journey at DevRev
- 1:46Pre-Istio network overview
- 3:12Post-Istio architecture benefits
- 3:53Why Istio service mesh
- 6:01Istio install and upgrade
- 7:45Automate data plane restarts
- 8:05Migrate to edge gateways
- 8:46JSON to gRPC transcoding
- 10:08Gateway timeout scaling
- 11:34Granular traffic management
- 12:58Debug connection closures
- 14:01Virtual service limits
- 15:04Secure clusters with Istio
- 16:50Istio authorization policies
- 17:12Monitor apps with Istio
- 18:56Reduce Prometheus load
- 19:38Monitor Istio control plane
- 21:03Scaling Istio observability
- 21:44Optimize Prometheus with Thanos
- 22:06Future multicluster Istio
- 23:10Custom operator for upgrades
related talks
36mIstio, we have a problem! Understanding and fixing bugs with a service-mesh
36mIstio: Real world scenarios
33mObserving and Troubleshooting your Microservices with Istio
33mPlatform Engineering meets Service Mesh: an SRE love story
24mIstio in Production: Expected Value, Results, and Effort
19mIstio, we have a problem! Understanding and fixing bugs with a service-mesh
12mThe Road Ahead: Istio Observability
46mService Mush: Debugging Istio Deployments