Kimi K2 and Our Contributions to Open Source
summary
Kimi K2 is an open-weight 1 trillion parameter foundation model that activates about 32 billion parameters per token, trains on 15 trillion tokens and uses 384 experts to keep individual expert matrices small. It releases in July and shows strong coding and agentic tool-call behavior. Core contributions include the moon optimizer (20–50% better token efficiency vs Adam with ~3% runtime overhead), checkpoint engine (pushes 1T weights to 256-GPU inference in ~20s), and decode context parallel (≈8× KV-cache reduction, 2–3× throughput).
AI-written summary · about
chapters (7)
- 0:00Kimi K2 introduction
- 2:52K2 capabilities and impact
- 7:30Moon optimizer for trillion models
- 12:30Checkpoint Engine middleware
- 16:02Decode Context Parallel overhead
- 19:12Other open-source contributions
- 21:01Q&A: Optimizer efficiency
related talks
33mFine-tuning a small model for style/vibe (a Kimi distillation and beyond)
35mUsing coding agents with open models
42mKeynote: LLM Dev Status in Japan
13mKeynote: To Nemotron and Beyond: Building the Open Source AI Ecosystem
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
17mOpen Innovation: Unlocking AI's Potential
49mOpen Models: The Secret Weapon for Next-Generation Software
17mMooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture...