Building next-gen speech synthesis for Bookmate audiobook service
summary
Baseline TTS trains on 90 hours of training data and deploys across more than 100,000 books; voice selection narrows ~50 candidates to two final voices recorded ~30 hours each; acoustic model uses Tacotron2-style encoder/attention plus pause tokens for prosody control. Next-gen stacks vector-quantized audio tokens with a GPT-like predictor and diffusion to generate mel-spectrograms, adds contrastive rescoring over 32 hypotheses, and claims quality wins (overall 64–36, dialogue 78–22). Engineering cuts training time with masking, dynamic batching and mixed precision to reduce training from 30 to 3 days.
AI-written summary · about
chapters (6)
- 0:00Next-gen speech synthesis intro
- 3:17User expectations for TTS
- 6:47Enhancing baseline TTS model
- 11:32GPT-like synthesis
- 19:11Faster training and results
- 20:42Q&A
related talks
35mSpeech Synthesis with Tacotron2 and PyTorch
22mDiscover Speech Synthesis and it’s Evolution
22mWhy TTS Models Now Look Like LLMs
42mdata.bythebay.io: Shubho Sengupta,
17mServing Voice AI at Scale
27mWhat happens when AI starts grocking streaming audio directly
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
43mTalk by Om Thakkar (OpenAI)