~/talk

Geometry of Efficient Fine Tuning: LoRA, Intrinsic Dimension, Subspace Learning

slides:none on file

Reproduces MNIST intrinsic-dimension results on a 784-200-200-10 MLP and shows D90 = 750 from ~200k parameters, meaning 750 learned directions hit ~90% of best accuracy. Implements random subspace training with projection matrix 156000×750 stored as PyTorch buffers and intrinsic params as nn.Parameters, and measures compute, walltime and accuracy versus LoRA. Demonstrates LoRA’s low-rank adapters (e.g., LoRA rank 3 ≈ 60× compression) and provides working code and a four-part blog series; experiments successfully reproduce the papers' claims.

AI-written summary · about

  1. 0:00Introduction to PEFT and LoRA
  2. 3:56Understanding intrinsic dimension
  3. 10:28Why networks are over-parameterized
  4. 16:15Subspace training in PyTorch
  5. 20:38LoRA and matrix decomposition
  6. 24:55Practical impact of intrinsic dimension
  7. 26:23Coding tools and process
  8. 28:36Frozen weights and gradient updates
  9. 30:22Training speed and convergence
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.