Klaviyo's Journey to Robust Model Serving with Ray Serve
summary
Klaviyo describes a production DART (DAtascience RunTime) model-serving platform built on Ray Serve and KubeRay, running two identical Ray clusters behind Route 53 weighted routing for failover. Deploys use templates, unit + integration tests (a fixture spins up Ray Serve), and Grafana metrics; request routing uses the power of two choices scheduler. For HA they back GCS with Elasticache Redis but warn don't share Redis; runtime pip installs fill ephemeral storage and head-node failures take down clusters. They plan single-cluster-per-app for stronger isolation.
AI-written summary · about
chapters (6)
related talks
25mRay Serve: Patterns of ML Models in Production
13mRay and Anyscale: An Optimization Journey
21mAchieving Scalability and Interactivity with Ray Serve
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
29mIntegrating High Performance Feature Stores with KServe Model Serving
31mHosting AI models with Ray and GKE
24mSkyRay: Seamlessly Extending KubeRay to Multi-Cluster Multi-Cloud Operation