Scaling the Ray Serve Controller to Thousands of Replicas
summary
Systematic scalability experiments push a single Serve controller to 2,048 replicas and break down where time goes: heavy cost comes from Pydantic serialization, Python time-series merging, and RPC-heavy metrics ingestion. Replacing validation with a Pydantic v2 migration, moving time-series aggregation to a Cython metrics aggregation path, switching RPC payloads to zlib L3 compression and throttling in-flight RPCs plus a sliding-window health-check reduces control-loop latency, yields ~7x faster loops, ~3.6x higher throughput, and ~15x fresher replica metrics.
AI-written summary · about
chapters (11)
- 0:00Introduction to Ray Serve
- 1:29How Ray Serve Cluster Works
- 3:14Serve Controller Basics
- 7:12Performance Bottlenecks Identified
- 7:35Benchmarking Setup
- 9:42Optimizing Pydantic Usage
- 12:09Autoscaler Algorithm Improvements
- 13:52Metrics Injection Optimizations
- 18:10Health Check Mechanism Updates
- 21:19Performance Improvements Results
- 22:01Future Work and Q&A
related talks
31mScaling Python to thousands of nodes with Ray
31mBeyond the Limits: Scaling Kubernetes Controllers Horizontally
23mImproving Ray for Large-scale Applications
47mMonolith to Microserv. at 20M+ requests per second: Latency and Scale Challenges
26mRay: A System for High-performance, Distributed Python Applications
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads
14mBuilding High Availability and Scalability Online Computing Applications on Ray
31mKlaviyo's Journey to Robust Model Serving with Ray Serve