Speeding up Whisper for speech recognition by 10x - Optimisations and walkthrough
summary
Claims a 10x speedup by combining engineering tricks and model changes and shows measured timings: a 10-minute sample runs in ~62s with fp16 baseline, ~60s using flash attention/SDPA, ~37.9s with speculative decoding, chunked batching can be 43.8s (worse with speculative in batches), and a distilled model plus fp16+SDPA+chunking hits ~17.2s. Uses Torch.compile/attention kernels, torchaudio preprocessing, chunking/batching, quantization paths (Q4/Q8) and recommends FlashAttention2 on modern GPUs.
AI-written summary · about
chapters (15)
- 0:00Introduction to whisper
- 3:00Why optimize whisper
- 4:25Measuring performance
- 5:41Initial optimization: fp16
- 7:40Demo: fp16 implementation
- 10:46Flash attention 2
- 14:12Demo: sdpa implementation
- 16:26Speculative decoding
- 21:38Demo: speculative decoding
- 23:32Parallel audio chunking
- 25:42Demo: parallel chunking
- 28:04Distil-whisper model
- 30:56Distil-whisper demo
- 33:23Summary and takeaways
- 35:24Q&A session
related talks
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
15mOptimizing inference for voice models in production
10mProfiling Whisper AI Model on RISC-V: CPU, GPU, and NPU Performance on the DC-ROMA AI PC
27mAccelerating Generative AI
08mHacks to Make LLM Training Faster
01h 16mOptimizing workloads with Transformers and Diffusers
11mOptimizing Model Inference with PyTorch 2.0
58mVoice Control Made Simple – Panel Discussion