Private Tuning of LLMs in Practice: From VaultGemma to Custom Fine-Tuning
summary
Claim: Differential Privacy is practical at production scale using JAX Privacy to train and fine-tune Gemma-family models including VaultGemma. Implementation uses DP-SGD with gradient clipping, noise addition, Poisson sampling; a 1B model trains from scratch with epsilon ≈2 using batch size ~520,000 for ~100,000 steps, and fine-tuning scales to ~30B with LoRA and ε=5, δ=1e-5. Empirical audits (credit-card and PII tasks, NP-extractability) show memorization drops under DP; tooling supports accumulation, efficient clipping, and common numerical pitfalls (FP16 noise bug) are documented.
AI-written summary · about
chapters (11)
- 0:00Private tuning of LLMs
- 1:54Data extraction threats
- 4:00Differential privacy basics
- 4:43DP-SGD algorithm
- 6:04JAX Privacy library
- 7:29VaultGemma 1B pre-training
- 9:57Fine-tuning needs DP
- 10:39Fine-tuning Gemma with DP-SGD
- 11:59Privacy auditing of DP models
- 15:09Training DP models advice
- 16:54Summary and resources
related talks
19mPractical Considerations for Differential Privacy
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
34mTalk by Amer Sinha (Google)
31mScaling privacy embedded distributed ML systems
01h 14mTutorial: Privacy, Part II
43mPrivacy Amplification by Sampling in Practice
01h 14mStatistical Inference and Privacy, Part III
01h 07mLarge-Scale Private Learning, Part II