Together Goes Brrr: Threading Research & Production with Torch Compile
summary
Deployment flow at Together AI uses torch.compile with the Dynamo tracer and Inductor backend to turn Python model code into profiled, fused kernels; automatic vertical fusion and epilogue merging remove redundant reads/writes across prefill and decode, raising arithmetic intensity for GEMM/GEMV; adaptive kernel generation across batch sizes, CUDA graph captures, optimized intra-node all-reduce and custom kernel registrations close remaining gaps. The result simplifies research→production for custom architectures and supports production inference at scale, serving billions of user queries.
AI-written summary · about
related talks
23mBlazing Fast GenAI Inference With Torch.compile
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
23mSeamless Integration: Custom Kernels in the Torch.compile Stack Wi...
30mOur Journey With TorchTitan
22mIncreasing the Impact of AI Through Better Software
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
10mLightning Talk: Faster Than SOTA Kernels in Torch.compile With Subgrap...
23mHow Pytorch Powers Training Inference