What happens when AI starts grocking streaming audio directly
summary
Compares a classic pipeline (STT → LLM → TTS) built on Jambones with streaming fixes for VAD and pipelining and a demo that runs at roughly two‑second to sub‑second first‑utterance latencies and works “okay,” against direct audio tokenization via Ultravox (Llama 8B → upcoming 70B, LiveKit SDK) that feeds audio tokens into the model. Claims function calling plus audio-first models reduce hallucinations and naturalness issues, but require keeping megabytes of audio history and massive streaming scale (hundreds of millions of user-hours).
AI-written summary · about
chapters (8)
- 0:00Introduction to audio AI
- 2:53Building a voice AI platform
- 7:06Challenges in audio conversational AI
- 13:26Pipeline approach demo
- 16:59Direct audio tokenization
- 20:10Speech-to-speech models future
- 23:19Market implications of audio AI
- 24:44Q&A session
related talks
22mVoice Agents That Think While They Talk
22mWhy TTS Models Now Look Like LLMs
30mOrchestrating Real-Time Multimodal AI Agents with Rust
58mMusic Generation via Masked Acoustic Token Modeling
01h 04m“Hello, AI?!” – Real-Time Voice Interactions with LLMs
01h 07mBuilding and Scaling an AI Agent Swarm of low latency real time voice bots
19mLLM-Enhanced Multimodal AI: Revolutionizing Audio/Video Interaction
15mOptimizing inference for voice models in production