Finetuning Stable Diffusion with low-rank adaptation techniques
summary
Claims that low-rank adapters reproduce near full fine‑tune quality while cutting memory and trainable parameters: Stable Diffusion’s UNet is ~860M params, full fine‑tune pulls optimizer state to ~9 GB and needs multi‑GB checkpoints, whereas using LoRA with rank 128 frees roughly 6 GB on a 16 GB GPU and yields adapter files from tens to hundreds of MB. DreamBooth gives best fidelity but creates multi‑GB checkpoints; hypernetworks only alter cross‑attention, remain checkpoint‑dependent, and mixable adapters enable new hybrid styles.
AI-written summary · about
chapters (21)
- 0:00Fine-tuning Stable Diffusion intro
- 2:35Generative models and diffusion
- 5:09Stable Diffusion internals
- 6:52Fine-tuning methods overview
- 7:55Textual inversion explained
- 11:03HyperNetworks for style/object
- 12:52Full vs partial fine-tuning
- 14:41Memory needs for fine-tuning
- 17:28Low-rank adaptation intro
- 19:13LoRA reduces trainable params
- 20:16LoRA image generation perf
- 21:41LoRA vs DreamBooth
- 22:26Mixing LoRA adapters
- 23:11LoHa and LoKr variants
- 24:56LoRA with Hugging Face
- 25:40Q&A: LoRA limitations
- 26:47Q&A: Forgetting and model size
- 28:16Q&A: Lightweight UNet options
- 29:18Q&A: Optimizers and trade-offs
- 31:04Q&A: Multi-GPU LoRA
- 32:07Closing remarks
related talks
12mEfficient Workflow Serving for Diffusion Models with Many Adapters
12mMemory-Efficient AI: How PEFT and PyTorch Enable Accessible LLM Fine-Tuning
31mLarge Scale Stochastic Training of Neural Networks
16mtorchtune: Easy and Accessible Finetuning in Native PyTorch
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
31mFine-tuning LLMs: A Cost-Benefit Analysis for Businesses
28m🧼 From GPU-poor to data-rich: data quality practices for LLM fine-tuning
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I