Squirrel - Efficient Data Loading for Large-Scale Deep Learning
summary
Open-source Python library Squirrel implements chainable iter streams, pluggable drivers and a catalog to realize a decentralized data mesh; it uses message pack and asynchronous IO so local image loading matches ffcv performance while remote S3/GCS reads are faster in preliminary benchmarks. It provides sharding hooks, caching and async prefetch to keep GPUs fed, supports multi‑modality and DALI offload, and ships as a library + plugins — the demos and benchmarks show it working end‑to‑end.
AI-written summary · about
chapters (14)
- 0:00Start
- 4:11Idealized data loading
- 4:53Large scale image datasets yield many problems
- 8:17Data Loading landscape
- 12:00Key Requirements What we learned the hard way
- 15:22Main components
- 16:35Streaming samples using Iterstreams
- 19:41Loading various data formats
- 20:54Custom data format
- 23:55Runtime transform accelerators
- 26:09Retrieve data from your catalog
- 26:42Data Source Sharing
- 29:24End-end distributed example
- 31:37Key goodies
related talks
22mBringing Deep Learning into the Big Data Analytics
29mStructuring Data for Scalable AI Applications
19mThe Small Model Infrastructure Nobody Built (So We Did)
14mSoftware 2.0 Needs Data 2.0: A New Way of Storing and Managing Data for Efficient Deep Learning
21mOptimizing Model Serving on Kubernetes With Model Streaming
49mDistributed Deep Learning
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
39mAccelerating Data Science with RAPIDS