Diffusion-Based Sound Synthesis in Music Production
summary
Claims use pretrained latent diffusion models (specifically AudioLDM 1 & 2) wrapped into a C++ JUCE plugin that calls a Python FastAPI inference server, and code is on GitHub. Evaluation with six participants finds device choice shifts outputs, best guidance scale ~4–5, and a sweet spot of 10–15 steps before quality degrades. Output is useful for exploratory, texture-style sounds but fails at precise single hits (e.g. 808 kick), runs at 16 kHz, and requires command-line setup for the server.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:05Motivation for diffusion synthesis
- 3:12Related work on synthesis
- 4:59Latent diffusion for sound
- 7:07Building AudioLDM instrument
- 8:08Live demo interface
- 11:10Evaluation and parameters
- 13:39Model limitations and improvements
- 16:33Creative use cases
- 18:46Live sound generation
- 20:55Q&A
related talks
50mAI'll Be Back: Exploring Diffusion Models for Image, Audio & Video
10mDiffusion models
19mYou Might Not Need 50 Diffusion Steps
51mThe Merger of Machine Learning, Music and Programming
10mLightning Talk: Diffusers: Bringing Cutting-Edge Diffusion Models to the Masses
09mComposing And Improvising Using Sound Content-Based Descriptive Filtering
30mProgramming for entertainment: Music and its perpetual inertia
04mThe Sound Sketchpad: Expressively Combining Large and Diverse Audio Collections