Hominis: Training Human-centric Large Language Models
summary
Hominis is a 13B model trained to match or beat Llama 2 while claiming 50x less carbon; training ran eight months for pretraining and four months for instruction tuning on the Leonardo supercomputer using roughly 32,000 GPUs. Developers used Red Pajama V2 plus a Hominis Reserve (GitHub and arXiv scraping), bf16 mixed precision (≈50% memory savings), FSDP, gradient checkpointing, a LoRA-or‑git workflow for low‑rank adapters, byte‑pair tokenization, and Q4m quantization for 87.5% inference memory reduction; model is Apache‑2.0 and available at hominis.io/playground.
AI-written summary · about
chapters (9)
- 0:00Hominis and human-centric LLMs
- 1:10Human-centricity in AI
- 3:22Transformer design for sustainability
- 6:55Model scale and hardware fit
- 7:59Dataset selection for training
- 9:03Training tricks for HPC
- 12:56Hominis 13B performance
- 13:37Extending to multimodality
- 17:10Q&A on training
related talks
48mBringing Llama 3 to Life
31mBuilding LLaMA: Meta's Director of GenAI
30mLearning to wield large language models
20mSelf-hosting AI LLMs - a beginners guide
18mSmolLM: The Rise of Smol Models
31mBuilding Large-scale Localized Language Models
31mDeploying Large Language Models with Ease
42mFine-tune and deploy LLMs from Hugging Face (AIM304)