Revisiting Scalarization in Multi-Task Learning
summary
Claims that linear scalarization cannot in general trace the Pareto front for linear multi‑task models, proving concretely that when hidden width q >= k the Pareto front collapses to a single point (q >= k yields single Pareto point), while in underparameterized regimes (q = 1 or q = k−1) the feasible set becomes a union of ellipsoids and is nonconvex unless the Gram or its inverse is elementwise nonnegative. Uses 2‑layer linear algebra, projection/hat matrices, and ReLU width constructions; shows randomized weighting helps and proposes Chebyshev scalarization with online mirror descent (primal GD + exponentiated dual) with an online‑to‑batch conversion; code is open sourced.
AI-written summary · about
chapters (17)
- 0:00Multitask learning intro
- 1:28Multitask feature basics
- 3:37Linear scalarization basics
- 6:14Linear scalarization limits
- 8:02Randomized weighting heuristic
- 11:42Adaptive weighting schemes
- 16:23Pareto front convexity findings
- 18:31Linear networks theory
- 23:34Over vs underparameterized
- 26:05Nonconvex Pareto fronts
- 30:30Nonlinear models convexity
- 34:02Randomization to convexify
- 35:26Chebyshev scalarization
- 37:32Minmax via online learning
- 40:47Online to batch conversion
- 41:50Scalarization methods comparison
- 42:13Summary and open questions
related talks
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
43mThe Linear Algebra of Deep Learning
23mOn Implicit Regularization in Deep Learning
46mKernel and Deep Regimes in Overparameterized Learning
01h 23mImplicit Regularization II
01h 02mTheory and Big Data
30mComputational Barriers For Learning Some Generalized Linear Models
01h 11mA Theory of Multi-objective Machine Learning