Heterogeneous Pipelines as the Unlock for ML at Scale
summary
Motive rebuilds its ML eval pipeline on Ray so each stage runs on a right-sized tier: CPU-heavy fetch/decode on wide cheap instances, GPU inference on a separately sized tier, and CPU fusion downstream. Disaggregation makes inference outputs cacheable inference boundary, raising GPU use to >90% GPU utilization and letting ~95% of eval iterations replay from cache for a 200x GPU-free eval speedup; talk includes Ray Data code for diagnostics, two‑dimensional parallelism, and identifying cacheable seams.
AI-written summary · about
chapters (10)
- 0:00Introduction to talk
- 2:50Problem with end-to-end testing
- 3:54Key ideas for scaling ML pipelines
- 5:21Pipeline design experiments
- 8:55Why modular design failed
- 11:45Optimal pipeline boundaries
- 15:37Caching model inference
- 17:03Linear scaling with Ray
- 18:29Combinatorial testing optimization
- 20:19Key takeaways and Q&A
related talks
29mScaling ML Pipelines for Autonomous Driving with Ray
28mScaling Multimodal Data and Reinforcement Learning with Ray
41mBoost ML on Heterogeneous AI Accelerator with Ray on K8s
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
26mData Processing on Ray
44mPanel: ML Platform on Ray
35mReddit's ML Evolution: Scaling with Ray and KubeRay
46mScaling ML Pipelines with Feast, Ray and Kubeflow