Writing Machine Learning Pipelines Against Object Storage
summary
Argues for building ML pipelines that read and write directly to object storage (MinIO) so compute only pulls batches instead of copying entire datasets, using TensorFlow for training and KubeFlow for orchestration. Demonstrates an IMDB sentiment pipeline that preprocesses to object storage, trains a 135k‑weight model, checkpoints and serves from the same store, and reaches about 85% accuracy. Emphasizes cheaper, scalable storage and easier automated retraining versus moving 1.31 TB datasets around.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:15Problem ImageNet
- 3:36Training vs Storage Infrastructure
- 5:24ML Pipelines
- 8:05Reusable Components
- 10:29The glue
- 12:31Solution: Leverage data in Object Storage
- 31:24Summary
related talks
35mWriting Machine Learning Pipelines Against Object Storage
24mObject Storage Driven Machine Language Workloads
31mBuilding a Data Infrastructure for AI/ML
33mCompliant Data Management and Machine Learning on Kubernetes
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
37mHow to Accelerate Model Training and Eliminate the I/O bottleneck for the Cloud
46mAccelerating Machine Learning DevOps with Kubeflow
41mInfrastructure for ML Applications