“Hello, AI?!” – Real-Time Voice Interactions with LLMs
summary
Real‑time LLMs accept raw waveform input (speech in, speech out), use WebRTC + getUserMedia for browser audio streams, and require a pub/sub session model (session.update, audio‑buffer events) instead of simple request/response. Demos show a working voice form filler that detects crash location from an image via tool calls, a color‑palette tool invoked as JSON, and a solar‑system controller; local offline setups run but produce poorer accuracy. Pricing is roughly $1/min or ~$32 per 1M input tokens.
AI-written summary · about
chapters (20)
- 0:00Real-time AI interactions intro
- 3:58Generative AI evolution
- 7:11From chatbots to multimodal
- 13:09What are real-time models
- 17:55Real-time model traits
- 20:24Real-time use cases
- 23:17OpenAI vs Gemini APIs
- 26:14WebRTC for real-time audio
- 29:07OpenAI API with WebRTC
- 32:00OpenAI console demo
- 36:10Session events explained
- 38:37Session and tools config
- 42:30Publish-subscribe model for real-time interactions
- 43:12MCP tool calling
- 45:40Adding tools to OpenAI API
- 48:48Voice solar system demo
- 54:33Real-time pricing
- 55:57Local real-time limits
- 59:17Summary and future
- 1:00:22Q&A
related talks
49mHello JARVIS - Building Voice Interfaces for Your LLMS
32mBuilding LLM Voice Bots with Open Source Tools
19mLLM-Enhanced Multimodal AI: Revolutionizing Audio/Video Interaction
22mVoice Agents That Think While They Talk
38mLeveraging LLMs to enhance your Audio/Video DApps
01h 02m[Full Workshop] Building Conversational AI Agents
19mAudio Streaming - Using WebRTC for building your own Voice AIs
39mLive voices with LiveView