MonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference...
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
MonoNN compiles an entire static neural network into a monolithic kernel on a single GPU to slash CPU-to-GPU launch and off‑chip traffic, empirically delivering an average 2.01× end‑to‑end speedup (≈2.0× at batch=1, ≈1.8× at batch=16/32) and up to 7.3× vs TVM, 5.9× vs TensorRT, 1.7× vs XLA on evaluated models and GPUs (T4, A10, A100). Key techniques are instruction rescheduling to trade ILP for TLP, on‑chip resource exploitation, global thread barrier merging, and schedule‑independent group tuning; compiler integrates with TensorFlow and source is on GitHub.
AI-written summary · about
chapters (12)
- 0:00MonoNN introduction and motivation
- 3:13Non-computational GPU overhead
- 5:00MonoNN compiler solution
- 7:32Monolithic kernel resources
- 9:18Monolithic kernel optimizations
- 11:45Context-aware rescheduling
- 14:38On-chip memory access
- 16:49Thread barrier merging
- 18:13Autotuning and codegen
- 19:16MonoNN performance
- 20:42Ablation study breakdown
- 21:24Conclusion and future
related talks
19mOptimizing CNN Model Inference on CPUs
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
26mNeuOS: A Latency-Predictable Multi-Dimensional Optimization Framework for DNN-drive
18mTC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs
42mAnima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor, CalTech
23mThe Race to the Bottom: Low Latency in the age of the Transformer
43mIntel Nervana Update + Productizing AI Research
14mKairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing