MLPerf: A Benchmark Suite for Machine Learning – David Patterson
summary
About 20 contributors from five organizations (including three universities and three industry firms) form a cooperative called MLPerf that builds a benchmark suite for machine learning with the explicit goals of fair comparisons, replicable benchmarks, and affordability for academia. It publishes full results every quarter, plans annual refreshes of workloads and datasets, closes loopholes that produced misleading performance in past suites, and emphasizes real end-to-end tasks (not synthetic rates) to drive investment decisions and accelerate progress.
AI-written summary · about
chapters (4)
- 0:00The history of benchmarks
- 3:14The rise of SPEC benchmarks
- 5:37Embedded computing failures
- 6:30Introducing MLPerf
related talks
39mMethodology of ML in Performance Testing (How to Find Bottlenecks Automatically)
01h 16mReproducible machine learning and science with python
29mEnhancing Performance Test Results with ML & LLMs: Communication Beyond the Numbers
26mMLbase: A Distributed Machine Learning System
19mApples, Oranges, and ML Models: Model Validation vs Benchmarking
31mMove beyond academia: Introducing an industry-first tabular benchmark
33mOpen Standards for Machine Learning Deployment
26mML Product Experiments at Scale