State-Space Models and Deep Learning: Is there a new revolution on our doorstep?
summary
Claims that state-space models give linear state transitions enabling parallelizable training plus fast O(n) inference for very long contexts; derives from Legendre-polynomial continuous SSMs discretized into convolutions whose kernels can span millions of steps; the hard engineering is matrix A design (diagonal A fails, special structure and fast multiplication matter); implementations like S4/S5 stack many SSMs and Mamba (S6) makes A context-dependent; works well for audio genomics and long video and matches small LLMs (~1B) but still trails large transformers on expressiveness.
AI-written summary · about
chapters (12)
- 0:00Introduction to speaker
- 2:50Transformers: pros and cons
- 5:05RNNs: revisiting limitations
- 8:21Introduction to SSMs
- 14:01Math behind SSMs
- 19:23Discretized SSMs as RNNs
- 22:27Convolutional representation
- 25:48Evolution of SSM architectures
- 28:27Mamba and selective SSMs
- 30:45Hardware optimization
- 32:40Applications and domains
- 35:19Q&A and conclusion
related talks
18mState Space Model Programming in Turing.jl
01h 04mNonlinear Model Reduction: Using ML to Enable Rapid Simulation of Extreme-Scale Physics Models
45mLarge language models are not a paradigm shift
41mSub-Linear Algorithms meets Large Language Models
28mLarge Multimodal Models: Understanding the State of the Art in Compute
01h 06mOn cognitive maps, LLMs, world models, and understanding
12mEmergent Trends in Large Multimodal Models
01h 00mReducing the Dimension of Language: A Spectral Perspective on Transformers