GraCE: Unlocking CUDA Graphs with Compiler Support for ML Workloads
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
GraCE is a compiler framework built atop PyTorch2 that unlocks CUDA Graphs for 25 ML workloads (TorchBench, HuggingFace, TIMM) and requires no code changes. It applies automatic code transforms to raise capture coverage (examples: from near-zero to ~99%), uses pointer indirection to avoid per-replay tensor copies (measured >99% data movement reduction), and runs offline cost–benefit profiling because 25% of naïve CUDA graphs hurt performance. Reported speedups include up to 3.34x and generation-specific gains (H100 1.5x, B200 2.4x).
AI-written summary · about
chapters (11)
- 0:00GraCE introduction
- 1:22CPU launch overhead
- 2:47CUDA graphs reduce overhead
- 4:32Static inputs challenge
- 6:59Precompiled kernels handling
- 9:06Data movement indirection
- 9:28Compiler transformations increase coverage
- 11:13CUDA graphs slowdowns
- 12:39GraCE’s three optimizations
- 13:00End-to-end performance
- 14:45Conclusion
related talks
12mCoding Agents for Compiler Construction: Beyond the...
45mOptimizing the Performance of Machine Learning in Enterprise Java SaaS with GraalVM, Python and CUDA
32mDeploying & serving optimized ML pipelines using graalvm
23mBlazing Fast GenAI Inference With Torch.compile
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
25m"Rust for AI & Accelerated Computing"
01h 00mBringing Clang and C++ to GPUs: An Open-Source, CUDA-Compatible GPU C++ Compiler
16mMirage: A Multi-Level Superoptimizer for Tensor Programs