Cross-Region Model Serving: PyTorch Inference, Observability...
summary
Production-tested architecture for running PyTorch inference across three regions handling 10K+ RPS per region with a P99 time-to-first-token target of <200 ms. Uses a mixed fleet: vLLM + Triton + Torch Compile orchestrated by KServe, Envoy L7 latency routing behind a global accelerator, warm pools and circuit breakers for N-1/N-2 failover. Observability relies on OpenTelemetry per-request spans, Prometheus/Grafana for TTFT and inter-token latency, and CI/CD with Argo CD, MLflow, bake-time canaries and automated rollback.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:04Multi-region inference need
- 3:31PyTorch serving evolution
- 5:37Inference request flow
- 8:27Multi-region deployment
- 9:51Observability for LLM inference
- 12:39Quality governance & rollback
- 13:43Key takeaways & tooling
- 14:27Q&A
related talks
46mServing machine learning models as an inference API in production
12mVertical Mobility: Inference from MVP to Trillion Parameters
51mSpeeding up deployment of machine learning models
22mDeploying computer vision models at scale devops-free
34mEfficient Resource Management
32mEnd-to-End Observability for LLM Inference: From Token To GPU
29mScaling to Millions of ML Models to Solve the Problems of SRE and Security
33mDealing with the versioning of production-ready models