Keynote: Foundation to Build LLMs
summary
Swallow is a family of LLMs created by continuing pretraining on Llama 2 and Mistral bases using compute on ABCI. The training used a cleaned Japanese corpus extracted from ~63.3B Common Crawl pages downsampled to ~2.7B then heavily filtered, produced models released on Hugging Face, and achieved about +7 points on Japanese benchmarks; they also expanded the vocabulary by ~11k tokens to roughly halve token length and double effective context capacity, and used model‑merge techniques to add chat/instruction abilities.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:28LLM development history
- 7:10LLM trends in Japan
- 14:39LLM research community
- 15:00SwallowLLM features
- 22:13Continual pretraining
- 27:59Vocabulary expansion
- 33:00Open LLM benefits
- 35:52Model merging techniques
- 36:13Lab LLM research
- 39:50LLM development summary
related talks
05mKeynote: Teaching LLMs New Knowledge
12mKeynote: How LLMs Have Changed the Way Organizations Do AI
15mKeynote: Common Corpus: Opening Data for Building Open Source LLMs
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
46mUnder the Hood of LLMs — all those questions you never dared ask about them
12mKeynote: The Llama Ecosystem: Past, Present and Future
44mKeynote Speaker - Simon Willison
28mThe Data That Shapes Foundational LLMs