Speeding up deployment of machine learning models
summary
Describes a production pipeline that turns annotated contracts into versioned model archives and deploys them via Argo Workflows and Git-driven merges. Models are served with TorchServe, Text Generation Inference, and vLLM, monitored with Grafana for GPU and latency, and code is standardized with init/preprocess/infer/postprocess classes so models can be swapped without rebuilding images. Memory-saving techniques include flash attention, LoRA, and quantization to run multi‑billion-parameter models across GPUs; structured JSON outputs plus lightweight LoRA tuning raise output validity from ~16% to ~99%, and drift monitoring triggers retraining when distributions or concepts shift.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:48ML model deployment at D Trust
- 8:11ML component breakdown
- 16:54ML model release pipeline
- 20:26Inference code standardization
- 26:31LLM inference tools
- 37:52Text Generation Inference
- 43:30Model monitoring in production
related talks
23mAccelerating Model Deployment with MLOps
21mMastering the DevOps Machine Learning Pipeline
56mDevOps for Machine Learning
35mHow to deploy machine learning models to production (frequently and safely)
31mMLOps In Practice – How To Run Your Machine Learning Models In Production At Enterprise Scale
17mContinuous deployment for ML: The new software development lifecycle
46mProduction Model Deployment
26mKubernetes and MLOps for Scalable and Reproducible Generative AI