Leveraging the Possibilities of Ray Serve
summary
Built a fully automated digital authentication service called validator that runs on-prem and processes front/back ID video plus face checks, using Ray Serve with a split CPU/GPU/API cluster split and Ray actors for async event synchronization. Migration from Kafka Streams proof-of-concept reduces frameworks and maintenance, disables unwanted library multithreading, uses custom resources, and enables model shadow testing. Measured outcome: about ~4x speedup in end-to-end processing and straightforward scaling and deployment.
AI-written summary · about
chapters (12)
- 0:00Introduction to Ray Serve
- 1:08What is digital authentication
- 5:05From proof of concept to scalable architecture
- 7:36Choosing Ray Serve over alternatives
- 10:11Implementing Ray Serve architecture
- 12:42Designing Ray Serve clusters
- 13:45CPU cluster optimizations
- 17:19GPU cluster setup
- 18:43API cluster for high availability
- 20:08Synchronizing tasks with Ray actors
- 24:44Starting the process flow
- 26:31Results and future plans
related talks
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
21mLeveraging Ray to Enhance Machine Learning Models in Finance
31mKlaviyo's Journey to Robust Model Serving with Ray Serve
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
24mBuilding Complex Data Analytics Pipelines with Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
24mDistributed Black-Box Model Explanation with Ray
30mHow to Scale Unstructured Data Processing with Ray Data