Power-aware Deep Learning Model Serving with μ-Serve
summary
microserve is a power-aware model serving system that builds a sensitivity profile per model partition, clusters partitions with similar frequency sensitivity, and applies sensitivity-based partitioning plus dynamic GPU frequency scaling to cut energy use. Evaluation on V100 GPUs and open-source LLMs shows 1.2–2.6× power reduction with SLOs preserved, 30–40% lower job completion time and up to 2.2–3.6× throughput improvement. It uses a lightweight BERT proxy predictor (≈7.6 ms overhead) with 5–6 output-length classes to avoid head-of-line blocking.
AI-written summary · about
related talks
16mSuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
37mAccelerate and Autoscale Deep Learning Inference on GPUs with KFServing
40mScalable machine-learned model serving
12mPreview: ML Serving
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
18mModel Serving for Deep Learning
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...