The Many Faces of Heterogeneity: Federated, Continual, and Modular Learning
summary
Claims that a single ridge-regression aggregator called Fed3R ridge regression recovers the centralized solution for linear probes and often outperforms FedAvg; a generalized heavy ball momentum that averages client updates over a window (optimal window ≈ 1/participation rate) cuts client drift and matches full‑participation convergence; a balanced replay buffer removes schedule variance in continual learning; and sparse low‑gradient fine‑tuning plus pruning reduces interference so independently fine‑tuned experts merge with less accuracy loss, provided they share a common pretrained initialization.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:06Data and task heterogeneity
- 2:45Continual learning basics
- 3:51Federated learning basics
- 5:24Model merging in async training
- 7:32Robust continual learning
- 9:52Transfer to federated learning
- 11:18Fed3R beats gradient methods
- 13:14Bias in federated momentum
- 16:04Merging for modular learning
- 21:50Composable models
- 25:29Interference-free merging
- 29:22Key takeaways and future directions
- 31:41Q&A
related talks
01h 05mTutorial: Federated Optimization, Part III
34mDecentralized Machine Learning
22mPanel Discussion
50mA Multi-Group Approach To Algorithmic Fairness
01h 00mFedOpt for LLMs
01h 07mIndustry Presentations
01h 21mMulticalibration: Towards Fair Decision Making
34mTraining without sharing: Is Federated Learning the answer to data privacy in AI?