Exact Unlearning of Finetuning Data via Model Merging at Scale
summary
Proposes SIFT-Masks, an exact unlearning method that constrains fine-tuning to agree with a shared global sign vector so local task updates are sign-aligned or zero, then applies locally computed masks that do not depend on the merged model. Merging up to 500 task models, it recovers per-task accuracy (5–80% lift over naive averaging), stores a single merged checkpoint plus masks, and achieves exact unlearning with up to 250x less compute than other exact-retraining baselines on multi-task text datasets like TOFU and Sent140.
AI-written summary · about
chapters (7)
- 0:00Collaborative learning challenges
- 4:29Exact unlearning in collaborative learning
- 8:25Model merging for unlearning
- 12:39Limitations of merging methods
- 16:04SIFT-Masks for exact unlearning
- 17:19Evaluating unlearning methods
- 24:05Conclusion and future work
related talks
30mUnforgettable, that's what you are: Evaluating Machine Unlearning and Forgetting
28m🧼 From GPU-poor to data-rich: data quality practices for LLM fine-tuning
36mLanguage Model Post-Training in 2025: an Overview of Customization Options Today
58mFederated Open Language Models: Training LMs on Distributed Data
32mThe Importance of Forgetting for Both Humans & Machines
01h 27mChallenges in Making LLMs Safe and Robust
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
20mKeynote