Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
summary
Paper proves that solutions satisfying dropout stability (and a stronger noise stability variant) are connected by explicit low-cost paths, constructed by alternating convex top-layer interpolations and rewiring moves; the proof uses induction on layers and yields paths with at most 10 line segments. A lower bound shows two-layer nets can have disconnected global minima when the number of samples exceeds neurons (H+2 examples). Constructions work in theory and in MNIST experiments but degrade on CIFAR-10 (VGG-11) with random 50% pruning.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:45Mode Connectivity[Freeman and Bruna 16, Garipov et al. 18, Draxler et al. 18]
- 3:14Outline
- 3:40Background: non-convex optimization
- 4:51Equivalent local minima and symmetry
- 8:56(Partial) short answer: overparametrization
- 13:42Deep Neural Networks
- 20:31What kind of local min are connected?
- 22:32Dropout stability
- 25:14High level steps
- 26:46Direct Interpolation
- 28:43Connecting a network with its dropout
- 30:07An example path for 3 layer network
- 30:54Example path explained (3) - (4)
- 35:10Noise stability
- 36:43Experiments
- 42:03Conclusions
- 42:40How can we use mode connectivity?
related talks
42mTowards Verification of General Neural Networks: A Dual Approach
45mFollow the Leader with Dropout Perturbations
28mTowards the Universal Law of Robustness for Deep Neural Networks
01h 03mUnderstanding ReLU Networks Through Graph Problems and Polyhedra
33mProvably Efficient Algorithms for Training Deep Networks
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
01h 04mRandomized Embeddings and Neural Networks
59mTutorial: Statistical Learning Theory and Neural Networks I