Voice Agents That Think While They Talk
summary
Pipeline uses WebRTC 20-ms frames for realtime audio, FFmpeg for transport, and VAD via Silero or Deepgram; latency targets are under 500 ms latency budget. Architecture streams ASR → token chunks → fast LLMspeaker for immediate replies while a background thinker or two-track LTS architecture runs heavier reasoning, sidecars, and a reducer to merge metadata. Uses async tool calls with filler speech to hide 500 ms–5 s ops and semantic triggers and sinker to decide when to sync state; claims this covers about 80–90% of cases.
AI-written summary · about
chapters (18)
- 0:00Intro to Voice Agents
- 0:50Why Voice Wins
- 1:55Voice vs Text Dynamics
- 2:44Core Pipeline Loop
- 3:33Streaming and Model Choices
- 4:51Realtime Audio Transport
- 5:42Turn Taking and VAD
- 9:01Latency Budgets and Tools
- 9:48Tool Calls Without Pauses
- 11:06Ecosystem and Platforms
- 12:04Speed vs Reasoning Depth
- 12:49LTS Dual Track Architecture
- 13:22Semantic Triggers and State
- 15:33Example Cancellation Flow
- 17:32Interruptions and Barging
- 18:54Sidecars and Reducer Models
- 20:07Freshness and State Control
- 21:13Wrap Up and Takeaways
related talks
30mOrchestrating Real-Time Multimodal AI Agents with Rust
27mWhat happens when AI starts grocking streaming audio directly
01h 04m“Hello, AI?!” – Real-Time Voice Interactions with LLMs
01h 58mWhat If Your Backend Could Talk? Building Real-Time AI Voice Agents
01h 07mBuilding and Scaling an AI Agent Swarm of low latency real time voice bots
01h 02m[Full Workshop] Building Conversational AI Agents
16mServing Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing
29mAndroid Gets a Voice: Building Real-Time Agents