Synthetic Gradients – Decoupling Layers of a Neural Nets

on YouTube ↗41m

" summary

Explains synthetic gradients as a method to break the sequential dependency imposed by backpropagation when training neural networks. Describes how learned gradient models can be used to decouple layers so they can be updated without waiting for true backpropagated errors, changing training flow for multilayer nets.

AI-written summary · about

" chapters (11)

1. 0:00Start
2. 0:24Outline
3. 1:38How do neural nets learn?
4. 2:28Quick Refresher on back prop
5. 4:39In Nutshell
6. 6:25Problem
7. 13:05How synthetic gradients are used
8. 14:45How synthetic gradients are produced
9. 20:37You have taken so many approximations, how do you know this does not affect the quality of training ??
10. 24:05Feed-forward Neural Nets
11. 25:04Complete Unlock

" related talks

• 52mSplitting Gradient Descent for Incremental Learning of Neural ArchitecturesQiang Liu · Frontiers of Deep Learning 2019
• 11mProxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient DescentSpencer Frei · Meet the Fellows Welcome Event
• 27mDemystifying Deep Neural NetworksRosie Campbell · Front-Trends 2017
• 50mIntegrating Constraints into Deep Learning Architectures with Structured LayersZico Kolter · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient DescentNadav Cohen · Frontiers of Deep Learning 2019
• 28mDemystifying Deep Neural NetsRosie Campbell · Render Conference 2017
• 45mExplaining Landscape Connectivity of Low-cost Solutions for Multilayer NetsRong Ge · Frontiers of Deep Learning 2019
• 04mNeural Networks Demystified: Part 5MLconf SF 2015