Federated, Synthetic, Personalized: Heterogeneity Here or There?
summary
Production on-device LMs use federated learning plus DP and secure aggregation to reach user-level epsilons mostly around 10 or lower and some under 1 by adding public pre‑training; a new BLT‑DP‑FTRL algorithm and SI‑CIFG architecture cut deployment cost and enable on‑device DP training. Synthetic data from public LLMs (including a conditional 100M‑parameter generator with topic histograms, filtering and reweighting) speeds convergence, improves AB test typing metrics, and is already used in deployed proofreading features.
AI-written summary · about
chapters (9)
- 0:00Federated learning intro
- 3:00Federated learning evolution
- 8:19Privacy in Gboard models
- 19:48LLM synthetic data
- 29:16Synthetic data for errors
- 34:54Synthetic data research
- 47:25Heterogeneity in learning
- 51:19Personalization future
- 52:40Q&A
related talks
48mSynthetic Data as an Enabler for Learning from Decentralized Private Data
01h 01mToward Provably Private Federated Learning
41mFederated Learning: Machine Learning on Decentralized Data
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
44mPersonalized Federated Training of Diffusion Models with Privacy Guarantees
42mFederated Learning: Unlocking the Values of Discrete Data for Enterprise
29mBalancing Thin Line Between Data Intelligence & Privacy
34mDecentralized Machine Learning