Voice fingerprinting & recognition
summary
Baseline uses wav2vec then moves to modified Whisper (OpenAI Whisper pretrained on ~6,680,000 hours) with injected voice fingerprinting layer plus pooling/aligning layers to produce joint diarization+ASR. System correctly detects four concurrent speakers in a meeting recording but overlapping speech causes transformer loops and invented tokens, so WER and diarization error remain only decent. Music-separation modules are reused to split voices into single-speaker channels; approach is open‑source based and robust to noise but not yet production‑grade.
AI-written summary · about
chapters (8)
- 0:00Voice fingerprinting intro
- 3:20Multi-speaker separation challenges
- 7:12Use cases & applications
- 10:04Historical speaker separation
- 12:32Baseline models & open-source
- 14:58Evaluating Whisper for voice
- 17:28Model limitations demo
- 19:15Future directions
related talks
25mBeyond Transcription: Building Voice AI That Understands Conversations
31mOff-the-shelf HuggingFace models for audio deepfake detection
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
45mAuditory Intelligence : beyond speech recognition
13mMining Parliamentary Gold - Building Hebrew ASR
22mBuilding next-gen speech synthesis for Bookmate audiobook service
16mThe End of Awkward AI Transcriptions
31m“Sorry, Could you repeat that again?” - Speech Recognition with Python