Software 2.0 Needs Data 2.0: A New Way of Storing and Managing Data for Efficient Deep Learning
summary
Converts large unstructured datasets into unified tensor arrays stored on object storage so training reads them like local numpy arrays, supports petabyte scale and dataset versioning, and streams to GPUs at ~875 MB/s over network matching local throughput while running 20x cheaper than in-memory approaches. Integrates with Ray to scale arbitrary transformations across a cluster (example: an Apache Beam job cut from ~500 CPU-hours to ~100 CPU-hours, ~5x faster). Benchmarked against TensorFlow Datasets+Ignite and used on 80M-document and multi-terabyte image workloads.
AI-written summary · about
chapters (6)
- 0:00Introduction to Software 2.0
- 3:53Introducing Data 2.0 standard
- 7:25Key challenges in data handling
- 10:55Integration with Ray for scaling
- 13:00Use cases and community growth
- 13:41Call to action and conclusion
related talks
29mStructuring Data for Scalable AI Applications
01mSOFTWARE 2.0 NEEDS DATA 2.0
31mBuilding a Data Infrastructure for AI/ML
39mUnlock the potential of your data: Optimize your data store architecture...
41mBig Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
32mAnyscale's Ray Data: Revolutionizing Batch Inference
59mSpark 2.0
24mObject Storage Driven Machine Language Workloads