Language Model Post-Training in 2025: an Overview of Customization Options Today
summary
Overview of concrete options and what actually works: standard SFT and parameter‑efficient methods like LoRA / QLoRA / OLoRA let you add small delta weights or soft prompts; continual learning via OSFT (adaptive SVD split) freezes top singular vectors and updates a low‑rank orthogonal subspace with an unfreeze rank ratio (0–1 example: keep top 80% frozen) to reduce forgetting without calibration data. Reinforcement methods use rollouts and a rules verifier—GRPO with rules‑based verifier—to assign 1/0 rewards and compute advantages. Training Hub provides pip‑installable Python functions to run SFT and OSFT today, with multi‑GPU and memory‑efficient init.
AI-written summary · about
chapters (12)
- 0:00Language model post-training intro
- 1:04Pre-training crash course
- 5:19What post-training enables
- 6:00Supervised fine-tuning basics
- 8:50Handling chat templates
- 11:22Continual learning and OSFT
- 16:59Parameter-efficient fine-tuning
- 21:20Reinforcement learning for models
- 27:22GRPO explained
- 28:46Other post-training techniques
- 29:08Training Hub library
- 32:41Q&A
related talks
42mPost-Training Small Language Models: the adventures of a practitioner
21mWhen Prompting Runs Out: Post-Training an Open Model
29mLocal LLM fine-tuning: a practical example
28mOptimizing Long-Tail and MoE Challenges in Reinforcement Learning with SGLang
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
01h 38mTraining Large Language Models: Practices and Research Questions
32mFine-tune LLMs from Kaggle Models using (Q)LoRA
35mLLM Customization