FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
summary
Introduces a BP-free federated finetuning pipeline that estimates gradients via perturbed inferences (forward gradients) and updates only small adapter modules, enabling on-device tuning of models up to LLaMA-7B. Claims include up to three orders of magnitude faster convergence and 4.6× memory reduction versus standard federated backprop, uses client-adaptive stopping to pick the number of perturbations, and a discriminative perturbation sampler to skip low-value probes; provides an MRM mobile inference engine with MPU support.
AI-written summary · about
chapters (15)
- 0:00FwdLLM introduction
- 1:50Federated LLM fine-tuning challenges
- 4:00Backpropagation limitations
- 5:26Forward gradient method
- 7:31Forward-only fine-tuning benefits
- 8:55Parameter-efficient fine-tuning integration
- 9:39Client workload adaptation
- 12:05Perturbation sampler design
- 13:53Evaluation setup
- 14:56Performance results
- 16:20Scalability with increasing client numbers
- 16:43Memory and energy efficiency
- 17:46Llama results
- 18:50Summary and future work
- 19:31Code and deployment
related talks
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
15mResource Multiplexing in Tuning and Serving Large Language Models
01h 00mFedOpt for LLMs
55mFederated, Synthetic, Personalized: Heterogeneity Here or There?
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
37mFATE-LLM: Empowering Large Language Models with Federated Learning
29mLocal LLM fine-tuning: a practical example
19mSystem for Scaling Large Language Models