Keynote: Common Corpus: Opening Data for Building Open Source LLMs
summary
pleias launches the Common Corpus as a reproducible dataset of 1 trillion tokens first release drawn from public-domain cultural heritage (libraries, national archives) with provenance tracked, permissive licensing, and partnerships with the French Ministry of Culture and Hugging Face. Engineering focuses on OCR correction using LLMs, deduplication, provenance-aware pipelines and ethical classifiers, and the project reports aiming for 2–3 trillion tokens by summer to enable training open LLMs.
AI-written summary · about
chapters (10)
- 0:00Open data for open source LLMs
- 2:27Copyright challenges in training data
- 5:01Building massive open corpus
- 6:27Common Corpus from cultural heritage
- 8:35Advantages of long-context data
- 10:01Ethical and OCR challenges
- 11:52Limitations of heritage data
- 12:36Expanding with open scientific data
- 13:39Filtering Common Crawl for licenses
- 14:23Open data for communities
related talks
42mKeynote: Foundation to Build LLMs
30mKeynote: LLM Research and Development at National Institute of Informatics
01h 30mPreparing Data for LLM pretraining using open source Data Prep Kit
15mOpenAssistant's Opensource LLMs Saga
05mKeynote: Teaching LLMs New Knowledge
14mKeynote: Building AI with Open Source & Hugging Face
44mKeynote: The Art of Language: Mastering Multilingual Challenges in LLMs
37mExpanding the Capabilities of LLMs Through Community Collaboration