What goes into the making of LLMs
summary
Claims concrete numbers: six models trained for 200B tokens with 0/25/50/75/90/100% code show 25% code gives the best natural-reasoning (+8.2%), world-knowledge drops as code proportion rises but cool down with 20% code recovers ~10.1%, and code-heavy models beat text models ~12× on code benchmarks. Experiments run at 470M and 2.8B params (bigger wins), code types include web, markup, code-adjacent, and synthetic (synthetic yields top gains), recommendation: balance + synthetic + code in cool down.
AI-written summary · about
chapters (8)
- 0:00Introduction to Language Models
- 0:18Importance of Code Data in Pre-Training
- 1:16Phases of Training Language Models
- 2:59Experimental Framework and Setup
- 3:30Impact of Code Data on Model Performance
- 4:53Code Data Proportion and Quality
- 6:58Key Findings and Recommendations
- 8:22Future Research and Final Takeaways
related talks
33mPractical Evaluation of LLMs for Code
40mLLM for Coding, the State and lnitiative
36mLLMs and AI agents demystified
27mBuilding Mini Minds Our Adventures in Teaching Small LLMs to Reason
44mProduction-Grade LLMs & AI Agents
01h 18mThe Enigma of LLMs: on Creativity, Compositionality, Pluralism, and Paradoxes
46mUnder the Hood of LLMs — all those questions you never dared ask about them
49mThe Building Blocks of Today’s and Tomorrow’s Language Models