TorchBench: Guarding the Performance of the PyTorch Ecosystem with Continuous Benchmarking
summary
Continuous benchmarking for PyTorch that runs a reproducible model suite (grown from ~50 to over 113 models) with a Docker image tied to each nightly release, fast single‑GPU CI tests that finish in under three minutes, and integrations with torch.compile/PT2 plus backends like TensorRT and inductor. Metrics include numeric comparison against eager float32, QPS/latency, peak CPU/GPU memory and FLOPs, profiler traces, automated regression alerts with bisection for commits, and noise control where ~95% of models show <5% variance.
AI-written summary · about
chapters (14)
- 0:00TorchBench introduction
- 1:06Why TorchBench created
- 3:17Growth and contributions
- 5:07Composability dimensions
- 7:15Performance metrics validation
- 9:25How to use TorchBench effectively
- 10:50Contributing models
- 13:43Generating model inputs
- 15:10Adding custom backends
- 16:17TorchBench infrastructure
- 17:44Collaborations and benchmarks
- 19:06Future enhancements
- 20:31Call to action
- 21:22Q&A
related talks
11mSotabench for Reproducible Research
38mSPONSOR WORKSHOP / Zhiyi Ma, Shagun Sodhani / Facebook
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
23mBenchmarking Machine Learning Workloads on Kubeflow
12mPyTorch Partner Collaborations
21mThe New Vision of Compiler-Accelerated
30mOur Journey With TorchTitan
13mLightning Talk: Challenges and Standardization in PyTorch Ecosystem...