Federated Open Language Models: Training LMs on Distributed Data
summary
SILO Open LM trains a 7B transformer on public Common Crawl (1T tokens) then has each data owner do isolated continual pretraining (≈50B tokens) using MoE-aware silo training (shared frozen layers plus per-silo FFNs), merges resulting models into a single MoE and trains only the router on small proxy subsets (≈5B tokens). Experiments show ensembling is a strong baseline, weight-averaging fails for many silos, and the merged model yields ~27% relative improvement vs the public model with big gains on MMLU biology, CS and BIG-Bench Hard while specialists still outperform on GSM8K and code.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 0:18Importance of data
- 1:28Proprietary data crisis
- 2:43Proposed research directions
- 3:31Data access spectrum
- 5:37Siloed model setup
- 7:15Limitations of federated learning
- 8:37Q&A on industry landscape
- 10:32Experimental data sources
- 12:52Review of prior methods
- 14:13Weight merging and ensembling
- 22:06MoEification technique
- 24:05Applying techniques to silos
- 25:42MoE-aware siloed training
- 30:44Proxy data for routers
- 34:21Router training and merging
- 36:14Experimental setup details
- 37:01Baseline results
- 42:09MoEification and baseline comparison
- 44:34Main results
- 47:58Analysis and findings
- 49:16Open problems
- 50:50Audience Q&A
related talks
46mFlexOlmo: Open Language Models for Flexible Data Use
18mSmolLM: The Rise of Smol Models
46mSpeeding Up LLM Training by Over 45%
30mLearning to wield large language models
01h 05mControlling distribution shifts in language models: a data-centric approach.
55mFederated, Synthetic, Personalized: Heterogeneity Here or There?
49mBen Hall & Barbara Fusinska
25mTraining Large Language Models in the Open