Open-Source and Science in the Era of Foundation Models
summary
Argues open access matters and backs it with concrete results: a position paper finds only 9 of 30 model reports disclose train/test overlap; builds MLAgentBench — an LLM-driven problem-solving agent evaluated on 13 ML tasks where Claude is best — and a 25-agent city simulator; creates 1,000 agents from 1,000 two-hour interviews whose survey accuracy is 69% versus humans’ 81%; develops a p-value test on open weights that detects fine-tunes (Micu, StripedHyena, Llama3.2) and shows pruning and decentralized $10B-scale training are feasible.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:53Access to models in research
- 7:14Building agents with APIs
- 15:35Simulation agents and digital twins
- 23:47Black-box access limitations
- 25:14Open-weight models and reproducibility
- 28:24Testing model independence
- 37:48Open-source scaling challenges
- 41:41Conclusion
related talks
21mDemocratizing AI with Open Source Multi-Agent Systems
32mThe Rise of AI agents and open sources Ai models
17mThe Future of AI, from an Open Source Perspective
01h 16mLifting the Hood on Big AI - The Future of Transparency and Accountability in AI
42mOpen Source LLMs — Powering AI Accessibility and Innovation
16mKeynote: Open Source for AI and AI for Open Source
37mIs Your AI Truly Open ? Cutting Through the Hype
36mInsights from The State of AI Report 2023