Voice In, Visuals Out: The Agony and the Ecstasy
summary
Claims that voice-in visuals-out is viable now with concrete tradeoffs and engineering fixes and demonstrates an agent that files a Linear issue in real time. Targets are strict: ~100 ms ideal, ~200 ms for conversational feel, P95s of 5,000–10,000 ms are unacceptable. Practical recipe: use very fast models (Haiku class or smaller), GPT-5 mini proved too slow in practice, send inference every 1–2 seconds rather than waiting for silence, hand off heavy work to bigger models, and rely on prefix caching to hit latency and cost targets.
AI-written summary · about
chapters (4)
- 0:00The case for visuals output
- 2:15Voice as a high-bandwidth input
- 5:16The challenge of latency
- 8:30Optimizing for speed
related talks
25mEngineering voice agents: Latency, quality, and scale
17mBuilding Effective Voice Agents
27mWhy ChatGPT Keeps Interrupting You
36mHow We Built a Voice Agent with Temporal
20m10 Things I Learned Building Voice Agents in 2025
17mReimagining Customer Experiences Integrating Voice into Digital Strategy
17mThe Voice-First AI Overlay: Designing Conversational Co-Pilots
15mOptimizing inference for voice models in production