Building a Low-Latency Voice Assistant leveraging Elixir and Membrane
summary
Built an Elixir media pipeline using Elixir and Membrane that ingests RTMP, runs VAD and a custom end‑of‑speech classifier (10 ms inference), streams LLM tokens to punctuation, and stitches incremental TTS chunks. Using open models—distill whisper for STT (~250 ms), yamama 3.17B with ~50 ms time‑to‑first‑token and 40 TPS, and Style 2 TTS (~500 ms to first audio)—latency drops from ~6–10 s to ~2–3 s; system works but end‑of‑speech accuracy (~60%) still causes interruptions.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:55First iteration challenges
- 3:44Latency goals and issues
- 5:11Streaming LLM responses
- 6:37Introducing Membrane
- 7:43Silence detection approach
- 9:52Latency breakdown
- 10:35Open source models
- 11:41Optimizing LLM speed
- 12:23Text to speech improvements
- 13:27User experience focus
- 14:10End of speech detection
- 15:56Voice activity detection
- 17:18Live demo
- 19:52Project learnings
- 20:54Q&A session
related talks
39mLive voices with LiveView
33mDesigning Voice Agents for Real Conversations
17mBuilding Effective Voice Agents
36mHow We Built a Voice Agent with Temporal
25mEngineering voice agents: Latency, quality, and scale
15mOptimizing inference for voice models in production
16mServing Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing
13mVoice In, Visuals Out: The Agony and the Ecstasy