Neural Turing Machines: Perils and Promise
summary
Neural Turing Machines (NTMs) are differentiable controllers plus external memory that learn algorithms like copying and balanced-parentheses; implemented in Theano/Lasagne and Go, they copy sequences up to 100 and generalize beyond training lengths. Extensions in the Nature paper produce the Dynamic Neural Computer with temporal links and allocate/deallocate memory, yielding near state-of-the-art on bAbI and passes 95% of bAbI tasks in many categories. They work but require huge parameter counts, are numerically unstable, and need tricks (gradient clipping, Graves’ RMSProp variant, Adam, curriculum learning) to train.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:53What's a Turing Machine?
- 1:20What's a Neural Turing Machine? Neural Network Controller
- 3:11Neural Turing Machines can... Learn simple algorithms (Copy, repeat, recognize simple formal languages...)
- 4:59Balanced Parenthesis
- 7:11Problems Architecture dependent
- 8:02Hard to train Numerical Instability
- 9:08Gradient clipping Limits training speed of parameters
- 11:27Attention to initialization Memory initialization extremely important
- 13:47Dynamic Neural Computers
- 14:58Architecture updates(1)
- 18:35Performance on Inference Tasks
- 20:06References
related talks
43mThe Biological Path Towards Strong AI
53mAn Integrated Cognitive Architecture
37mNeural Networks are Programs, Too
12mComputation with sequences of neuronal assemblies
48mKeynote – Sameer Singh
45mThe Conscious Turing Machine (CTM), a formally defined Theoretical model of Consciousness
39mComputational Thinking on Learning Models
32mHow Neuroscience Makes AI Intelligent