Music Generation via Masked Acoustic Token Modeling
summary
Combines parallel iterative decoding with discrete acoustic token modeling to generate and infill music audio; uses SoundStream and a hierarchical residual VQ‑VAE (about 14 layers) to split coarse versus fine tokens, then samples coarse first and fills details. Training uses LoRA to fine‑tune on small palettes — roughly 10 minutes of target audio + ~6 hours on an RTX 3090 — enabling style transfer, audio compression, and masked‑token prompting. Demonstrations produce titi‑monkey‑like choruses from 10 minutes of nature footage; interactive real‑time control is feasible but currently experimental.
AI-written summary · about
chapters (14)
- 0:00Start
- 1:20Coordinated vocalizations
- 2:143 main patterns of interaction
- 5:02Many species have rhythmic vocalizations
- 5:38Temporal structure
- 9:50Making music together helps...
- 16:56The original DALL-E
- 18:36Autoregressive image generation
- 19:08Autoregressive Sampling
- 23:48MusicLM: Generating Music From Text
- 28:15VampNet: Masked Acoustic Token Modeling
- 33:52Developed for musical creation
- 34:20LORA fine-tuning for sound palette transformation
- 55:30What's next
related talks
33mContext-Aware Masking for Controlled Music Generation in MIDI
51mThe Merger of Machine Learning, Music and Programming
27mWhat happens when AI starts grocking streaming audio directly
22mWhy TTS Models Now Look Like LLMs
14mLightning Talk: Enabling the Audio Modality for Language Models
46mMusic Generation with Magenta: Using Machine Learning in
34mModelling emotional nuance of musical performance
22mBuilding next-gen speech synthesis for Bookmate audiobook service