Building and Scaling an AI Agent Swarm of low latency real time voice bots
summary
Builds a real-time audio-in/audio-out agent using Deepgram for low‑latency STT/TTS, configurable LLMs (OpenAI GPT‑4o shown), and NodeJS client/server with GitHub repos provided; demo places drive‑thru orders via function calling to an add_item API and returns spoken responses, using linear16 chunks and 12 TTS voices. Claims sub second latency end to end with options to self‑host for ~50 ms STT, and design patterns to scale to 1000s or millions of concurrent conversations with regional Kubernetes autoscaling.
AI-written summary · about
chapters (17)
- 0:00Building AI voice agents
- 4:37Real-time voice bots evolution
- 7:09Live demo: voice AI agent
- 9:02Workshop environment setup
- 12:14Voice agent API
- 17:46Backend API deep dive
- 22:54Voice customization & models
- 27:17Handling interruptions
- 30:59Voice agent approaches
- 33:08LLM-agnostic function calls
- 34:12Long-running function calls
- 37:56Monitoring & moderation
- 41:28Language & voice options
- 44:36Scaling for call centers
- 52:45Multi-agent swarms
- 56:18Infrastructure scaling
- 1:02:19Future of voice AI & Q&A
related talks
32mBuilding LLM Voice Bots with Open Source Tools
01h 58mWhat If Your Backend Could Talk? Building Real-Time AI Voice Agents
17mServing Voice AI at Scale
29mAndroid Gets a Voice: Building Real-Time Agents
30mOrchestrating Real-Time Multimodal AI Agents with Rust
22mVoice Agents That Think While They Talk
23mBuilding with OpenAI: What's Ahead
16mServing Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing