Why Number of Parameters Isn't a Good Performance Indicator for Live AI Processing
summary
Argues that parameter count and FLOPs poorly predict real-world speed, showing correlations with mobile latency are much lower than expected and memory access, branching and synchronization dominate runtime differences across CPUs, GPUs and NPUs. Recommends measuring latency and throughput on the target device, using techniques like reparameterization, knowledge distillation and quantize to FP16/INT8, and deploying with TensorRT, TensorFlow Lite, PyTorch Mobile or ONNX; gives a production anecdote where a simple linear regression outperforms a complex model.
AI-written summary · about
chapters (5)
- 0:00Introduction to AI efficiency
- 3:55Why model metrics matter
- 9:20Flaws in efficiency metrics
- 14:41Reliable performance measures
- 19:45Q&A and closing remarks
related talks
18mDeploying ML solutions with low latency in Python
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
25mLifecycle of a Parameter
30mUnderstanding GPUs
22mCentimani: Enabling Fast AI Accelerator Selection for DNN Training with a Novel...
21mThings I learned running neural networks on microcontrollers
19mReal-Time AI: Low Latency Solutions for Interactive Applications
14mOptimize your TensorFlow Lite models