Synthetic Data as an Enabler for Learning from Decentralized Private Data
summary
Centralized training on differentially private synthetic data using the Private Evolution pipeline (nearest‑neighbor histograms with DP noise, foundation-model seed generation and mask‑filling/diffusion variations) matches or slightly outperforms DP‑federated learning on small LLM finetuning while achieving 100x communication reduction and 6x lower client compute (ε≈1–1.3); theory shows Wasserstein‑1 error falls with n but needs exponentially many samples in ambient dimension, and preference‑based fine‑tuning narrows the utility gap.
AI-written summary · about
chapters (7)
- 0:00Synthetic data for private learning
- 2:40Private evolution algorithm
- 12:16Synthetic data vs federated learning
- 26:17Data contamination in evaluations
- 29:24Convergence of private evolution
- 43:30Preference optimization for synthetic data
- 45:18Future directions and challenges
related talks
01h 14mStatistical Inference and Privacy, Part III
55mFederated, Synthetic, Personalized: Heterogeneity Here or There?
33mPrivacy of Decentralized Machine Learning
45mPractical Experience with Making Synthetic Data Differentially Private
30m🤯 No data? No problem! Synthetic data to the rescue
13mDPSynth: From Research to Production—Engineering Differentially Private Synthetic...
10mWhen Privacy Guarantees Meet Pre-Trained LLMs: A Case Study in Synthetic Data
14mEstablishing Privacy Metrics for Genomic Data Analysis