Designing Voice Agents for Real Conversations
summary
Production voice agent code using Pipecat plus Daily (WebRTC) and any STT/TTS (Deepgram AssemblyAI Whisper) demonstrates concrete fixes: streaming pipeline, WebSocket orchestration, audio chunking, backpressure and echo cancellation, plus interruption handling that cancels TTS/LLM within ~50 ms. Targets a 200 ms latency budget while noting typical cascaded cloud setups hit ~1,100–1,300 ms and best measured public cascades ~755 ms; co‑located GPUs can reach ~500 ms. Turn detection options range from Silero VAD to open Smart Turn with 58.9% recall 68.4% precision and LLM choices like Nemotron‑3 Ultra (529 ms P50) or GPT‑4.1 (536 ms P50).
AI-written summary · about
chapters (17)
- 0:00Voice agent turn taking
- 1:06200ms constraint in voice
- 4:01Pipeline architecture components
- 5:28Silero VAD silence detection
- 8:37STT service turn detection
- 10:04VAD plus Smart Turn model
- 11:51Handling user interruptions
- 13:17Code comparison turn detection
- 15:09Latency in voice pipelines
- 17:17Benchmarking fast LLMs
- 19:47Production issues and UX
- 21:11Deployment and scaling
- 21:53Demo setup
- 23:17Demo Silero VAD
- 26:25Demo STT turn detection
- 29:43Demo Smart Turn model
- 32:00Summary and Q&A
related talks
17mBuilding Effective Voice Agents
25mEngineering voice agents: Latency, quality, and scale
20m10 Things I Learned Building Voice Agents in 2025
01h 10mFull Workshop: Realtime Voice AI
27mHow to Build Your First Real Time Voice Agent in Python...
36mHow We Built a Voice Agent with Temporal
27mWhy ChatGPT Keeps Interrupting You
01h 26mBuilding voice agents with OpenAI