Tricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
summary
WhisperSpeech is a MIT‑licensed open source TTS trained on tens of thousands of hours using Whisper for semantic tokens, Meta Encodec for compressed audio and Vocos for vocoding; training pipeline scales from small experiments to 80k+ hours on the JUWELS A100 cluster via LION. Data engineering uses WebDataset shards (5 GB) to get 300 iters/sec versus SSD stalls; faster‑whisper plus batching and the small English model reaches ~768× real‑time for transcription. Issues include misaligned timestamps and noisy chapter headers—fix by trimming first/last 30s—final inference hits ~4× real time on a 4090.
AI-written summary · about
chapters (10)
- 0:00WhisperSpeech intro and scaling
- 3:33Genesis and open-source vision
- 9:31Massive unstructured speech data
- 13:26Accelerating Whisper transcription
- 15:55Fixing Whisper transcription errors
- 19:26Iteration speed and scaling
- 23:16Scaling initialization pitfalls
- 26:46GPU acquisition for training
- 28:11Open-source release and community
- 30:00Future WhisperSpeech plans
related talks
43mSpeeding up Whisper for speech recognition by 10x - Optimisations and walkthrough
33mTranslate tech talks using open-source models and cloud-native Wasm
01h 34mReinventing Speech-to-Text Transcriptions with Go and Whisper
34mText-to-Speech Data Preparation and Fine-tuning Workshop
15mOptimizing inference for voice models in production
22mBuilding next-gen speech synthesis for Bookmate audiobook service
24mTowards Large-scale Speech Analytics Systems
14mLightning Talk: Enabling the Audio Modality for Language Models