Tabbers - turn every guitar playing video into a guitar lesson
summary
Takes an iPhone guitar performance with noisy, polyphonic audio and distant video and produces time-aligned tablature and an overlayed lesson-ready video. Combines audio cleaning and channel separation (Librosa, Demucs-style models), BasicPitch model for polyphonic note detection, computer vision (Ultralytics + MediaPipe + OpenCV) for fretboard finger detection, plus heuristics to handle fretboard tilt, slides/bends and asynchronous audio/video alignment. A Flask/FastAPI backend and Next.js frontend serve generated tab overlay on video; results are usable for learners after iterative fusion and rule-based cleanup.
AI-written summary · about
chapters (14)
- 0:00Welcome and intro
- 1:26Two key messages
- 2:12From raw video to guitar lesson
- 3:55Why guitar tabs are hard
- 5:00Music theory basics
- 7:27Existing tech limitations
- 8:33Audio processing with data
- 10:24Merging audio and video data
- 11:47Everything is data
- 12:08Cleaning audio signals
- 13:34High-res audio insights
- 14:35Video processing fixes
- 15:16Final algorithm tweaks
- 15:39Results and platform demo
related talks
55mInventing Guitaraoke: A Tale of Tech, Bugs, and Rock'n'Roll
26mHow to teach your computer to play guitar
01h 04mKeynote 3. Interactive generative models for music
31mBuilding a JavaScript Video Synthesizer for Live Performance
41mMusic and Machine Learning
31mHow to Use Machine Learning and .NET to Write Crushing Metal Riffs
30mPoseDance: Build a TikTok Trainer
43mHigh performance video editing for the mobile web