Lightning Talk: Backpropagation-Free Optimization in PyTorch
summary
Compares four concrete families — difference target propagation, direct feedback alignment, local layerwise losses, and forward-forward — and shows PyTorch-level patterns for explicit update loops without autograd, what learning signal each passes, what intermediate state must be stored, how parameters update, and what bottlenecks limit accelerator scaling. Demonstrates working code and benchmarks via a bio torch wrapper on convolutional datasets, notes methods save memory but struggle to form deep hierarchies and remain untested on large models. Highlights Pepita forward-forward pass and feedback alignment with fixed random matrices.
AI-written summary · about
chapters (9)
- 0:00Gradient-free optimization intro
- 0:41Evolutionary weight optimization
- 1:45Hebbian learning without BP
- 3:56Local updates and memory
- 6:22Limitations of BP-free methods
- 7:24Advantages of local rules
- 8:29Feedback alignment baseline
- 9:52Bio-torch library intro
- 10:58PEPITA training method
related talks
19mDIY PyTorch - How to Learn Deeply
01h 00mBackpropagation and Deep Learning in the Brain
12mLightning Talk: Building Intermediate Logging for PyTorch
19marchive – Soumitra Pandit
43mExploring Deep Learning Framework PyTorch
41mReally Deep Neural Networks with PyTorch
08mSponsored Session: Lightning Talk: PyTorch in Production: Boosting LLM Training and...
47mGet to Know PyTorch - A Cutting Edge Open Source AI Framework for Deep Learning