Channel Gating for Cheaper and More Accurate Neural Nets
summary
Introduces per-filter channel gating trained with Gumbel-softmax, an L0 sparsity term and a novel batch-shaping loss that matches the empirical CDF of gate outputs to a target (e.g., 50% on / 50% off) so gates avoid trivial or random behavior. A 20× wider ResNet-18 with these gates matches ResNet-50 while using far fewer MACs; TimeGate halves MACs on long-range video by skipping irrelevant snippets; continual-learning uses task-conditioned gates, freezes used filters and employs generative replay for task ID.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:58Babaks background
- 4:25What is conditional computation
- 8:19Conditional computation techniques
- 9:24Overarching motivation
- 10:35Different approaches
- 13:19Gating individual filters
- 15:25Cross entropy loss
- 19:55Bad shaping
- 21:41Bad gate shaping
- 25:05Bad gate shaping applications
- 25:44Continuously learning applications
- 32:39How are you allowing the task to choose the filters
- 37:35Long range activities
- 43:04Conditional gating
- 45:47Uniform sample
- 46:22Other applications
- 47:13Specific results
- 48:08Interpretation results
- 49:54Conditional gated networks
- 50:54Requirements for channel gating
- 52:25Path towards making channel gating generally applicable
related talks
34mGraph Neural Networks for Classical and Quantum Channel Coding
01h 59mSpiking Neural Networks
27mGAZELLE: A Low Latency Framework for Secure Neural Network Inference
16mGRACE: Loss-Resilient Real-Time Video through Neural Codecs
44mRobust Estimation and Generative Adversarial Nets
38mCorinna Cortes, Head of Research, Google, NYC 2017
01h 01mML for discrete optimization: Theoretical foundations
27mPattern Analysis: Learned vs. Engineered