Scaling Data Beyond English: Multilingual Scaling and Data Mixing Laws
summary
Paper runs 800 pretraining runs from 10M–8B parameters on MADLAD-400 and derives a cross-lingual transfer matrix and a bilingual transfer score that quantify which languages help Thai (and others). Effective data D composes monolingual tokens, a small set of close-transfer languages with tau weights, and an “other” bucket; per-language lambda models diminishing returns from repeated epochs. English is broadly helpful but asymmetric, script beats family at 2B scale, and beyond an inflection point pretraining from scratch can beat fine-tuning a multilingual checkpoint.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:25Scaling limits for multilingual models
- 3:58Low-resource language training
- 5:48Bilingual transfer efficiency
- 10:52Cross-lingual synergy matrix
- 17:37Script versus language family
- 20:44Multilingual scaling and mixing laws
- 28:28Fine-tuning vs pre-training
- 30:38Curse of multilinguality
- 34:11Summary and future directions
- 35:02Q&A
related talks
31mBuilding Large-scale Localized Language Models
45mHow multilingual is your NLP model?
44mTransfer learning in NLP
28mFine-Tuning Small Language Models with Python
22mMultilingual embeddings to scale NLP models to multiple...
24mEfficient Transfer Learning
01h 38mTraining Large Language Models: Practices and Research Questions
47mAya: Expanding the Worlds AI serves