How we brewed the Gigachat Pretrain data
summary
Describes how pretraining datasets for GigaChat are prepared: mapping web-data clusters across Russian and English segments, deduplicating massive volumes, classifying content by training value, and running experiments to evaluate data quality. Also covers technical challenges of clustering embeddings at billion-item scale.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:51Training data composition
- 4:45Data duplication methods
- 9:23Data filtering quality
- 12:37Data clustering
- 16:32Data quality evaluation
- 20:32Synthetic data generation
- 27:33Math problem generation
- 32:29Conclusion
- 33:11Q&A
related talks
35mHow We Launched YandexGPT
27mBuild a ChatGPT RAG Data Pipeline
01h 03mWhat comes after ChatGPT? Vector Databases - the Simple and powerful future of ML?
41mCloning ChatGPT with Hugging Face and Elasticsearch
01h 22mWho needs ChatGPT? Rock solid AI pipelines with Hugging Face and Kedro
37mHow ChatGPT Got Me Drunk
31mMachine Learning-Enabled Chatbots
32mFeed generation at ShareChat