Accelerate and Autoscale Deep Learning Inference on GPUs with KFServing
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Thursday, 2020-11-19· at 15:45· Intrado Virtual Event Platform,·
summary
KFServing v1beta1 plus Knative autoscaling and activator deploys TensorFlow/PyTorch BERT on GPUs using TF-Serving or Triton, enabling 20x speedup versus CPU and hitting 15 ms on V100 for a 24-layer FP16 BERT where CPU runs take ~2–3 seconds. Uses multi-model co‑location, model repo versioning, batching, shared GPU/shared-memory or binary payloads (binary gives ~17x speedup), and sets containerConcurrency=1 so Knative scales pods by observed concurrent requests.
AI-written summary · about
chapters (5)
- 0:00Start
- 20:27KF Serving Inference Protocol - Version 2 • Why a standard protocol? Inference clients can talk to multiple servers, increase portabilty Inference servers expand client base, increase utility
- 22:01Core Protocol and Extensions • Core protocol required for all conforming servers
- 26:23Inference • Send input tensors to specific model and get back output tensors
- 33:02High-Performance HTTP/JSON • HTTP/REST easy to use, but poor performance encoding tensors with JSON • Triton binary data extension resolves
related talks
39mIntroducing KFServing: Serverless Model Serving on Kubernetes
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
40mServing Machine Learning Models at Scale Using KServing
36mAccelerating High-Performance Machine Learning at Scale i...
30mAccelerate Federated Learning Model Deployment with KServe (KFServing)
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve
24mKServe Next: Advancing Generative AI Model Serving
25mMaking the Most Out of Your Hardware Accelerators in a Kubernetes Cluster