How Cookpad Leverages Triton Inference Server To Boost Their Model S...
summary
Cookpad converts PyTorch models into a Triton-ready package, uploads the directory to S3, and deploys via CI/CD so multiple models run concurrently on one GPU instead of one GPU per model. Triton provides dynamic batching and model instances and spreads replicas across GPUs; testing shows roughly 10x throughput and 10x lower latency versus a CPU service while using about 50% of a GPU at baseline, cutting inference cost and improving utilization.
AI-written summary · about
chapters (8)
- 0:00Start
- 4:53Running Ml in Gpus
- 5:57What Happens if I Want To Deploy a New Model
- 6:33What Happens if Your Model Requires Gpu for Inference
- 10:51Model Instances
- 17:20How To Deploy It on Triton
- 21:55Resource Utilization
- 24:33Fortica Scaling
related talks
25mMoving Inference to Triton Servers
22mDeploying computer vision models at scale devops-free
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve
51mNvidia Triton Inference Server: building production ML without developers
16mLightning Talk: Triton Compiler
25mRace to Superintelligence: Building Low-Friction AI Infrastructure for a Heterogen...
25mLeveraging AI Ecosystem for Portable and Sustainable GPU Kernels in HPC
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler