Accelerating On-Device ML Inference With ExecuTorch and Arm SME2
summary
ExecuTorch plus Arm SME2 runs SqueezeSAM on a single CPU core and cuts FP16 end to end latency from >1s to ~300ms yielding 3.9x end-to-end speedup. The pipeline uses PyTorch → ExecuTorch → XNNPACK → Arm Clyde AI SME2 kernels and evaluates FP16 and INT8. Operator-level profiling shows convolution speedups shift the bottleneck to data movement ≈40% of runtime / ~100ms. A practical fix is to delegate BatchNorm into XNNPACK, reducing transfer overhead by about 70% and unlocking further gains.
AI-written summary · about
chapters (7)
- 0:00On-device ML with ExecuTorch
- 1:24Vision workloads: SAM case study
- 4:39PyTorch to hardware execution
- 5:43Squeeze SAM performance
- 7:55Data movement overhead causes
- 9:43Optimizations across models
- 10:47Key takeaways and future work
related talks
27mDeploying GenAI for Audio Generation on Mobile CPUs With ExecuTorch
22mIntroducing ExecuTorch from PyTorch Edge: On-Device AI...
27mAccelerating Generative AI
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
16mAccelerating Web AI on
25mExecuTorch 1.0: General Availability Status for Mobile and Embedded...
16mHardware acceleration for on-device Machine Learning
33mExecutorch: Enabling and Accelerating PyTorch Models on Edge Devices