Real-Time AI: Low Latency Solutions for Interactive Applications
summary
Practical steps to cut interactive AI latency: distill a giant model into a task-specific student (example given from a Llama 60B teacher to a smaller assistant), apply quantization or run QU 0.5B / Phi3 tiny / Gemma on-device, and move inference to edge hardware or LPUs/TPUs. Use batching, parallel pipelines, embedding caching and async RAG. A pharmacy WhatsApp bot handling ~2,000 daily requests drops perceived waits from ~30s to near real‑time after pipeline tweaks and deploying Nvidia Jetson containers.
AI-written summary · about
chapters (9)
- 0:00The necessity of speed in AI
- 1:48Compressing AI models
- 6:43Edge computing benefits
- 8:20Specialized hardware
- 11:01Pipeline and cache optimization
- 14:05Real-world case study
- 15:37The impact of time on AI
- 16:20Summary and best practices
- 17:29Q&A session
related talks
31mLow-Latency Solutions for Resource-Constrained Environments
46mSmall AI is the next Big Thing
25mMachine Learning is Going Real-Time
36mApplied AI on the edge
48mThe Power of Edge AI: Revolutionizing Future of Real-Time, Autonomous Processing
18mDeploying ML solutions with low latency in Python
31mMy AI is slow. Make it faster!
29mGenAI in Backstage - Turbocharge Your Developer Productivity!