Moving Inference to Triton Servers
summary
Moves inference onto Triton Inference Server, splitting pre/postprocessing onto CPU nodes and running compiled models on GPU so a single G4 instance replaces older P2 instances (from ~$0.90/hr down to ~$0.50/hr) and one GPU serves multiple models (about four in this deployment). Adds a model and dataset registry with versioning, stores inference metrics and timings in a DB, compiles PyTorch models to TensorRT and reports up to 6x speedup in inference latency in production.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:17GPU inference challenges
- 5:01Restructuring production pipeline
- 5:43ML service architecture
- 9:14Centralized tracking benefits
- 9:35Triton Inference Server overview
- 11:45Resource optimization with Triton
- 12:50Deploying models on Triton
- 16:43Cost savings with G4 instances
- 18:08Getting started with Triton
- 19:11Summary
- 19:32Q&A
related talks
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve
32mHow Cookpad Leverages Triton Inference Server To Boost Their Model S...
22mDeploying computer vision models at scale devops-free
51mNvidia Triton Inference Server: building production ML without developers
25mRace to Superintelligence: Building Low-Friction AI Infrastructure for a Heterogen...
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
46mServing machine learning models as an inference API in production
23mThe Race to the Bottom: Low Latency in the age of the Transformer