VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
VTC introduces virtual tensor representations that map logical indices to multiple physical buffers to eliminate intermediate data-movement kernels, implements an automatic strategy via a VTOG and a greedy pointer-graph construction for an NP-hard selection problem, and integrates with existing kernels by only changing I/O. Implemented on PyTorch 2 and evaluated on A100/H100, VTC outperforms industry compilers (up to 1.93× speedup, 60% memory savings; 1.28× and 17.5% on average), and removes KV-cache update kernels and temporary allocations in Llama-3 decoder layers.
AI-written summary · about
chapters (12)
- 0:00VTC introduction and motivation
- 1:03Compute vs memory bandwidth gap
- 2:06Data movement overhead
- 3:09Llama 3 decoder layer example
- 4:12Virtual tensor optimization
- 6:20Comparison with existing frameworks
- 7:22Virtual tensor I/O benefits
- 8:46Virtual tensor profitability
- 9:48Opportunity graph construction
- 11:33Evaluation setup and benchmarks
- 12:58Comparison with VLM and fusion
- 14:23Summary and outlook
related talks
18mTC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
18mCoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
24mVerified Tensor-Program Optimization Via High-Level Scheduling Rewrites
28mFunctional Sparse Tensor Compilation
34mBeyond Transformers - Intro to RWKV Architecture & The World To...
28mC2TACO: Lifting Tensor Code to TACO