LiveMLP: ML Platform for Assisting Contact Center Agents in Real-Time
summary
A real time ML platform is built to attach stateful workflows to each call, route per-intelligence processors via a scheduler + supervisor, and keep latency bounded while scaling horizontally. Implementation uses Python microservices, ASR feeding per-call Redis streams, Kafka-style many-to-many messaging with a Redis sidecar to reuse connections, GPU-batched inference and Kubernetes hooks for graceful shutdown/state requeue. Load testing with Olaf load generator reaches 10,000 concurrent calls and shows stable latencies and improved business metrics.
AI-written summary · about
chapters (11)
- 0:00Start
- 6:08What do we want to answer?
- 10:50Design Pattern: Scheduler-Agent-Supervisor
- 13:00Platform Design
- 15:05Tech Stack
- 16:25Inside ML Processor
- 18:00Load Balancing
- 21:56Autoscaling Shutdown
- 23:48Ungraceful Shutdown
- 28:16Capacity Estimation!
- 29:16Key Takeaways
related talks
24mUnder Pressure: Applying ML in Real Time
40mBuilding highload ML powered service
32mHarnessing Large Language Models for Enhanced Customer Experience
31mUse-Case Driven ML Platforms
23mML Platform Challenges & Applications at Freshworks
38mReal Time ML Pipelines in Multi-Tenant Environments
11mScaling Conversational AI in Production
27mML Platform Journey