Workshop: LLMs Fine Tuning and Inferencing Using ON...
summary
Fine-tuning and inference use ONNX Runtime with Azure Container for PyTorch, AzureML, DeepSpeed stage 2 and LoRA on Mistral 7B v0.1. Training on 8x V100 32GB shows throughput rising from ~1.52 to ~1.76 iterations/s (~14%), and a 10k-iteration run drops from ~6600s to ~5750s (~12.5% time savings). Inference requires a 15–20 minute one‑time convert+optimize step, yields ~30 ms token generation and significantly faster prompt processing that avoids PyTorch OOMs at long sequence lengths, often >40% over torch compile.
AI-written summary · about
chapters (13)
- 0:00Fine tuning with ONNX runtime
- 5:19ONNX runtime overview
- 10:00ONNX runtime performance
- 12:00DeepSpeed GPU memory
- 13:24LoRA efficient fine tuning
- 15:08Azure Container PyTorch
- 17:56Mistral 7B model
- 21:35ONNX runtime training code
- 33:21ONNX runtime inference code
- 43:22Training performance results
- 45:48Inference performance results
- 51:47Azure ML fine tuning UI
- 55:48Q&A
related talks
08mHacks to Make LLM Training Faster
01h 18mOptimizing LLM Training and Inference Performance on GPUs
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
24mLLM Inference Optimization From Simple To Sophisticated
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
18mDeploying ML solutions with low latency in Python
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
38mOptimizing LLM Inference for the Rest of Us