Running gRPC at Scale: Lessons From the Frontlines of Production
summary
Practical, production-tested advice for running gRPC at scale: Kubernetes DNS and sidecar meshes (Envoy) for service discovery, Prometheus metrics plus Jaeger traces and structured JSON logs for P50/P95/P99 visibility, and client code patterns (round-robin DNS resolver, retries with exponential backoff — max 4 attempts, 0.2s initial backoff capped at 2s) shown as real production-ready examples. Guidance covers HTTP/2 long-lived connection hotspots, connection pooling and periodic resets, client-side load balancing, and careful connection draining and rollout strategies.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:53Production vs development
- 9:56Service discovery
- 13:07Observability in gRPC
- 17:26Load balancing
- 20:15Load balancing code
- 22:43Backward compatibility
- 24:55Deployment strategies
- 25:57Key lessons
related talks
01h 03m.NET gRPC - deep dive
37mHTTP and JSON for your gRPC Services
46mPracticalities of Productionizing Distributed Systems
05mProject Lightning Talk: Essential Resources for gRPC Development
05m⚡ Lightning Talk: Increasing Platform Reliability and Performance in One Week with a Service Mesh
41mKubernetes the Very Hard Way
38mTempering our Production Practices
24mHow to Be 10x SRE? A Deep Dive to Prometheus Operator