Cost and Performance Optimization of LLM Inferencing.
summary
Claims that avoid buying big GPUs: use prompt engineering and caching to cut calls to expensive models, plus cheap-model-first cascading and vector similarity search to return cached or small-model answers instead of full LLM runs. Typical cold starts take ~90 seconds so vendors keep containers warm ~15 minutes; firon says time slicing GPUs lets inferencing start in milliseconds by caching models in GPU memory and sandboxing with WebAssembly. The net: many requests never hit a costly large model, reducing real cost.
AI-written summary · about
chapters (16)
- 0:00Introduction and speaker background
- 1:04Why optimize LLM inferencing costs
- 2:51Workload types and real-time needs
- 4:58Prompt engineering before fine-tuning
- 7:05Hosting your own model considerations
- 8:10Over-provisioning and scaling challenges
- 10:58Latency and scaling thresholds
- 12:40Cost impact of vendor infrastructure
- 14:09Peak-to-average workload patterns
- 15:56Vendor cost structures and efficiency
- 16:40Cheapest inference: avoid unnecessary requests
- 17:45Caching and cascading model approaches
- 19:11Optimizing with vector databases
- 20:15Firon’s serverless GPU solution
- 21:39Call to action and resources
- 21:59Demo: LLM Explorer tool
related talks
30mLLMs: Beyond the Hype - A Practical Journey to Scale
34mOptimizing AI Inference with ML Compilers & Hardware
24mLLM Inference Optimization From Simple To Sophisticated
38mOptimizing LLM Inference for the Rest of Us
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
19mServerless inferencing: an infrastructure point of view
42mStop Treating LLMs Like REST APIs
30mAccelerated LLM Inference with Anyscale