Keynote: Marin: An Open Lab for Frontier AI
summary
Marin runs model development in the open using GitHub issues to pre-register experiments and pull requests that contain full experiment pipelines in Python so anyone can see every hyperparameter, data recipe and the trace from raw data through tokenization to evaluation. Rigorous benchmarks show claimed optimizer speedups shrink from ~40% to ~10% at scale, a data-efficiency study finds up to 5x reduction in required data via better tuning and ensembling, and open runs produced an 8B model that beats Llama 3.18 and a 32B run rescued by adding QK norm.
AI-written summary · about
chapters (13)
- 0:00Marin and open AI development
- 1:49Shift from open research to closed systems
- 3:15Open source AI and open development
- 5:22Marin’s mission for frontier models
- 7:06Marin’s open-source infrastructure
- 8:51Transparency in model training
- 9:32Benchmarking optimizers on Marin
- 10:15Data-efficient training algorithms
- 10:58Training frontier models from scratch
- 12:23Debugging 32B model training
- 13:28Community and Marin’s leaderboard
- 14:52Scaling laws and method validation
- 15:15Conclusion and call to action
related talks
16mKeynote: Accelerating AI/ML Development with Open Source Software
11mKeynote: Open Source Stack Empowers the AI Era- 16mKeynote: Accelerating AI/ML Development with Open Source Software
40mKeynote: Open-source Multimodal AI in the Wild
13mKeynote : The Open Source AI Alliance
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
01h 02mJoint Keynote Address: Scaling AI Sustainably: An Uncharted Territory
07mKeynote: Trusted AI: Bringing Trust Back into AI through Open Source