Cachew: Machine Learning Input Data Processing as a Service
summary
Cachew is a managed service that disaggregates TF Data Service pipelines into dispatcher and stateless workers, adds a caching layer, and applies auto-scaling and auto-caching policies by measuring node-level throughput and projected cache read times. It decides which hint locations to materialize, scales workers until fetch latency no longer improves, and shares cached preprocessed data across jobs. Evaluation shows up to 4.1x reduction in training time and 1.1x–3.8x reduction in cost.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:24Data Processing in ML Workloads
- 1:29Input Preprocessing Challenges
- 2:36Opportunities
- 2:59Current Landscape in ML Preprocessing
- 5:03Main Contributions
- 5:19System Architecture
- 6:06Client-Service Interaction in Cachew
- 7:03Main Features of Cachew
- 7:19Code Example
- 9:27Graph Representation
- 13:34Evaluation: Autoscaling Policy
- 16:12Evaluation: Autoscaling & caching for Multiple Tenants
related talks
14mHyCache: Hybrid Caching for Accelerating DNN Input Preprocessing Pipelines
41mBig Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
40mScalable machine-learned model serving
20mMArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving
20mTectonic-Shift: A Composite Storage Fabric for Large-Scale ML Training