Writing Machine Learning Pipelines Against Object Storage
summary
Argues that storing data in an object store and letting each pipeline stage read from it avoids repeated full-data transfers, enables burstable compute, and separates cheap storage from expensive GPUs. Demonstrates with MinIO + TensorFlow, using the IMDb sentiment dataset (a demo model of ~135k weights, ~85% accuracy) and notes large-scale context like 1.31 TB ImageNet / 14M images. Builds stage-based pipeline (extract, preprocess, train, eval, serve), keeps checkpoints and models in object storage, and runs it end-to-end on qflow and TensorFlow Serving successfully.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:15Problem ImageNet
- 3:36Training vs Storage Infrastructure
- 5:24ML Pipelines
- 8:05Reusable Components
- 10:29The glue
- 12:31Solution: Leverage data in Object Storage
- 31:24Summary
related talks
35mWriting Machine Learning Pipelines Against Object Storage
24mObject Storage Driven Machine Language Workloads
33mDeploying and Scaling Spark ML and Tensorflow AI Models
31mBuilding a Data Infrastructure for AI/ML
13mMLOps Pipelines to Deliver Tf.Lite Models to Distributed Compute Pools at the Edge
46mAccelerating Machine Learning DevOps with Kubeflow
57mMachine Learning Data Pipelines
52mMachine Learning Data Pipelines