Fine-tuning a small model for style/vibe (a Kimi distillation and beyond)
summary
Claims that a Kimi K2 vibe can be transplanted to a much smaller model are tested by distilling from a 1T-parameter Kimi into a 1.5B Granite 4 Hybrid using PEFT/LoRA, BF16 safetensors, and a custom pre-batching framework (cauldron_train) run on a 3090 rented via Vast.ai. Results show some checkpoints produce brief and punchy system prompt outputs and notably punchier resumes, but persistent looping issue, hallucinations and an uncovered bug leaving important Granite layers untrained mean the distillation is promising but imperfect.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:27Choosing base model
- 4:54Early training results
- 6:59Training framework issues
- 8:23Hardware and bug struggles
- 10:53Future plans
- 13:47Opening the floor for questions
- 14:36How fine-tuning works
- 22:01Training time investment
- 26:44Model checkpoints demo
- 30:58Live demo setup
- 32:01Closing remarks
related talks
26mKimi K2 and Our Contributions to Open Source
15mScaling the Next Paradigm of Heterogeneous Intelligence
31mFrontier results, on device
36mWhy Models Break Your Pipelines
20mEverything I Learned Training Frontier Small Models
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
42mPost-Training Small Language Models: the adventures of a practitioner
31mBuilding High-Quality & Trustworthy FMware