Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing
summary
Deploys Orpheus (Emotive, Realtime TTS) on GPUs with batch inference and LoRA clones, using Orpheus (Emotive, Realtime TTS) and vLLM with FP8 dynamic quantization. Engineering achieves ~105 tokens/sec on stock voices and ~95 tokens/sec for LoRA clones with batch size 10, while removing 600 ms head‑of‑line silence via fine‑tuning drops P50 latency to ~100 ms. Uses a consistent hash ring, Redis for routing, L40s/H100 GPUs, and the system runs in production handling multi‑GPU load.
AI-written summary · about
chapters (10)
- 0:00Introduction to Gabber and Real-Time AI
- 2:15Gabber's Mission for Consumer AI
- 4:17The Orpheus Voice Model
- 5:43Challenges in Voice Cloning
- 7:44Latency Management and "Head of Line Silence"
- 11:07Infrastructure for Batch Inference
- 11:36Leveraging vLLM and Dynamic Quantization
- 13:21Load Balancing with a Consistent Hash Ring
- 14:17System Architecture Overview
- 15:07Conclusion and Open Source Shout-outs
related talks
01h 07mBuilding and Scaling an AI Agent Swarm of low latency real time voice bots
33mDesigning Voice Agents for Real Conversations
30mOrchestrating Real-Time Multimodal AI Agents with Rust
15mAI Trends: Insights
22mVoice Agents That Think While They Talk
33mTranslate tech talks using open-source models and cloud-native Wasm
25mEngineering voice agents: Latency, quality, and scale
15mAI Trends, and Running LLMs in the Enterprise