Sotabench for Reproducible Research
summary
Sotabench benchmarks open source models by connecting repositories to continuous integration with GitHub so each commit can run reproducible tests; add one bench file and use framework‑independent libraries or the PyTorch helper. The beta runs include 150 ImageNet models and plot accuracy versus predictions/sec, exposing speed/accuracy tradeoffs and showing reproduce marks where repo results match paper claims. The service integrates third‑party tools and provides free GPUs for public repos.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:39Papers with Code
- 2:23Leaderboards
- 3:24Mission
- 5:24How it works
- 6:19Results
- 8:17Conclusion
related talks
25mTorchBench: Guarding the Performance of the PyTorch Ecosystem with Continuous Benchmarking
38mSPONSOR WORKSHOP / Zhiyi Ma, Shagun Sodhani / Facebook
08mLightning Talk: Standardizing CPU Benchmarking with TorchBench for PyTorch...
25mReproducibility in Embedding Benchmarks
18mSuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation
23mBenchmarking Machine Learning Workloads on Kubeflow
32mReproducibility in Embedding Benchmarks
31mMove beyond academia: Introducing an industry-first tabular benchmark