Geometry of Efficient Fine Tuning: LoRA, Intrinsic Dimension, Subspace Learning
summary
Reproduces MNIST intrinsic-dimension results on a 784-200-200-10 MLP and shows D90 = 750 from ~200k parameters, meaning 750 learned directions hit ~90% of best accuracy. Implements random subspace training with projection matrix 156000×750 stored as PyTorch buffers and intrinsic params as nn.Parameters, and measures compute, walltime and accuracy versus LoRA. Demonstrates LoRA’s low-rank adapters (e.g., LoRA rank 3 ≈ 60× compression) and provides working code and a four-part blog series; experiments successfully reproduce the papers' claims.
AI-written summary · about
chapters (9)
- 0:00Introduction to PEFT and LoRA
- 3:56Understanding intrinsic dimension
- 10:28Why networks are over-parameterized
- 16:15Subspace training in PyTorch
- 20:38LoRA and matrix decomposition
- 24:55Practical impact of intrinsic dimension
- 26:23Coding tools and process
- 28:36Frozen weights and gradient updates
- 30:22Training speed and convergence
