Building High Availability and Scalability Online Computing Applications on Ray
summary
Framework that turns Ray jobs into low‑latency, high‑availability online services by running a long‑running Ray cluster with a long‑running APP master actor that spawns trigger and worker actors, a dispatcher for dynamic load‑balanced task routing, and an external serving keeper proxy that manages cross‑cluster deployment and discovery. Uses external state persistence and in‑place updates to hit online SLA targets (aiming for 99.9% availability); thousands of upgrades run daily with most finishing in 5–10 minutes. Scale achieved: >20,000 CPUs, >3,000 device ports; autoscale latency remains an open challenge.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 1:19Problem analysis and Ray benefits
- 2:25System design on Ray
- 4:15Other online scenarios with Ray
- 5:39Challenges in high availability serving
- 7:05Serving keeper architecture
- 9:10Cross-cluster service discovery
- 10:17Online upgrades and SLA
- 11:22Scalability and auto-scaling
- 12:03Conclusion and future plans
related talks
15mRay: A System for Distributed Applications
26mRay: A General Purpose Serverless Substrate?
29mAdopting Ray @ LinkedIn
26mUsing Ray On Large-scale Applications at Ant Group
24mDistributed Black-Box Model Explanation with Ray
24mRay: A System for High-performance, Distributed ML Applications
31mManaging Large-Scale Ray Deployments: Cloud, On-Prem, and Kubernetes
42mRay: A distributed system for emerging AI applications