WhAM: Towards A Translative Model of Sperm Whale Vocalization
summary
A generative pipeline converts arbitrary audio into sperm-whale–style codas using a VampNet-style audio-to-audio model with a Descript VQ‑VAE tokenizer and a BERT-like masked transformer, fine-tuned via low‑rank adaptation on Watkins and Dominica coda cuts. Outputs audio-to-audio coda translator and embeddings sometimes fool experts, and objective Fréchet audio distance drops (beeps ≈0.9 → ≈0.5), but outputs show DC offset, added background noise and unnatural click onsets, and an AVES embedding outperforms WhAM’s for several classification tasks.
AI-written summary · about
chapters (14)
- 0:00WhAM project introduction
- 1:20WhAM model goals
- 2:02Sperm whale analysis limits
- 4:16Acoustic translation definition
- 6:31WhAM model architecture
- 9:25Neural audio codec details
- 13:44Data curation process
- 18:45Listener study evaluations
- 23:02Translation results
- 30:38Audio similarity measurement
- 37:47Embeddings for downstream tasks
- 45:16Ablation studies
- 46:19Future directions
- 48:49Q&A
related talks
01h 04mGPT-Whale Language Models at Tools for Structure Discovery and Hypothesis Generation
01h 03mGenerative AI and What Is Meaningful Sperm Whale Communication
53mAI as a tool for discovery in animal communication
16mThe End of Awkward AI Transcriptions
30mSound Modelling - parametric methods and deep learning...
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
33mTranslate tech talks using open-source models and cloud-native Wasm
20mVoice fingerprinting & recognition