Lightning Talk: Accelerated Inference in PyTorch 2.X with Torch...
summary
Torch‑TensorRT uses PyTorch 2.x Dynamo to convert models into TensorRT engines either just‑in‑time via torch.compile(model, backend="tensorrt") (handles arbitrary Python control flow, automatic graph recompile guards, and claimed Stable Diffusion <1s inference) or ahead‑of‑time via torch_tensorrt.compile(model, inputs=example_inputs) and DynamoDoTrace/DynamoDoCompile to produce serializable TorchScript/exported programs with shape ranges for dynamic inputs. Both paths perform tracing, lowering, partitioning and conversion to TensorRT, expose precision and min‑block‑size tuning, and plan fuller dynamic‑shape and precision support soon.
AI-written summary · about
chapters (12)
- 0:00Torch-TensorRT intro
- 1:04What is Torch-TensorRT
- 2:29JIT vs AOT workflows
- 3:12Shared optimization path
- 4:17JIT compilation workflow
- 5:44Conditionals in JIT
- 7:48AOT compilation workflow
- 8:10Exporting PyTorch models
- 9:33Dynamic shapes in AOT
- 10:16AOT serialization
- 11:40JIT and AOT similarities
- 12:03Summary and future
related talks
08mPyTorch 2.0: TorchDynamo
21mThe New Vision of Compiler-Accelerated
13mPyTorch 2.0
29mKeynote
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
36mDL Compiler Panel Discussion
43mExploring Deep Learning Framework PyTorch
32mKeynote: Fireside Chat with Soumith Chintala, Co-Creator of PyTorch & Merve Noyan