Serverless inferencing: an infrastructure point of view
summary
Demonstrates a live deploy of Lama 3.1 on A100 with autoscaling from one to three instances and a benchmark that triggers new VMs every 15 seconds, claims real time inferencing needs end-to-end optimization, measures CPU cold start via memory snapshotting at ~100–200 ms, cites PCIe Gen5 theoretical 500 GB/s, and argues software and alternative chips reduce waste and cut inference cost by ~10x; shows scale mechanics including scale to zero and model hot‑swap tradeoffs.
AI-written summary · about
chapters (9)
- 0:00AI infrastructure demand
- 3:40AI infrastructure types
- 5:47Serverless inferencing challenges
- 8:16Live demo: autoscaling endpoint
- 10:00Hardware efficiency in AI chips
- 12:16Software optimizations
- 13:42Distributed inferencing
- 16:53Low-latency inference optimization
- 17:37Future efficiency gains
related talks
24mCost and Performance Optimization of LLM Inferencing.
13mTailor-Made Inference at Scale
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
39mBuilding model testing infrastructure and scaling AI predictions
12mVertical Mobility: Inference from MVP to Trillion Parameters
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
24mChallenges with Ultra-low Latency LLM Inference at Scale
37mFrom Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes