Synthesizing Video and Turning it into Music!

on YouTube ↗09m

" summary

Uses WebGL and the Web Audio API to generate frames on the GPU, reads pixels off the GPU back into JavaScript, then sums columns into audio samples so each video row maps to time samples. Works at about 1k audio-frame callbacks giving ~43 frames per second against 44,100 Hz audio, so surprising frequencies like ~43 Hz and single‑cycle waveforms matter. Converts Adventure Kid dotwave files into images and even loops a cropped baby vocal sample to produce audible, working tones despite readPixels latency.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 0:35Webgl
3. 1:13Web Audio
4. 5:01Single Cycle Waveforms

" related talks

• 48mMaking Realtime Audio-VisualsMathieu Henri · JSConf.Asia 2015
• 33mCoding ArtMathieu Henri · Render Conference 2017
• 31mBuilding a JavaScript Video Synthesizer for Live PerformanceZachary Berry · JSConf US 2018
• 45mArt.jsMathieu Henri · ffconf 2016
• 43mUsing a modern web to recreate 1980S horribly slow and loud loading screensRemy Sharp · FrontEnd Con 2018
• 19mGenerative music, playful visualizations and where to find themLisa Passing · Web Audio Conference 2018
• 41mOrchestrating the Web Audio APISrikumar · JSFoo 2015
• 30mUsing a modern web to recreate 1980s horribly slow & loud loading screensRemy Sharp · JSConf EU 2018