FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
summary
FlexServe is a lightweight Python library that deploys PyTorch models as REST endpoints, avoiding ONNX/TensorFlow conversion by running multi-model ensemble logic inside one container so models can share GPU memory and respond in a single request. It uses Flask + Gunicorn, accepts batches from one up to practical HTTP limits, and returns per-model JSON predictions. Ensemble policies (majority, threshold) let endpoint sensitivity change at runtime without retraining. It proves faster than routing to separate endpoints and requires far fewer dependencies than KFServing or TorchServe.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:34Scenario
- 1:26Motivation
- 2:03Challenges
- 4:00Deployment Options
- 6:12FlexServe Features
- 7:44FlexServe Architecture
- 11:26FlexServe Batch Size
- 13:01Ensemble Features
- 14:34Conclusion
related talks
41mRESTful Machine Learning with Flask and TensorFlow Serving
39mIntroducing KFServing: Serverless Model Serving on Kubernetes
37mAccelerate and Autoscale Deep Learning Inference on GPUs with KFServing
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
16mSuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
42mDeploying ML at Scale with Kubernetes and TFX
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
40mServing Machine Learning Models at Scale Using KServing