“Many-model” Time Series Forecasting: Scaling PyTorch Training Across 1000s...
summary
A pipeline that trains thousands of PyTorch models in parallel by partitioning Snowflake tables, converting partitions to Ray datasets, and mapping each to a Ray actor; demo uses a 150k-row synthetic retail set (10 stores × ~500 products) to train per-store LSTMs, scale nodes live, and log partitioned models to the Snowflake model registry. Built-in retries, OOM detection that kills and surfaces failing partitions, and near-linear runtime reduction versus serial runs are shown. Ray scheduling + Snowflake data connector and partitioned model registry per key
AI-written summary · about
chapters (15)
- 0:00Many-model forecasting intro
- 1:05Complex time series forecasting
- 3:13Scaling micro-segmented models
- 5:02Many-model framework intro
- 6:04Architecture overview
- 7:29Partitioning and orchestration
- 8:54Demo setup and data
- 12:29Training function definition
- 14:18Dynamic cluster scaling
- 15:21Real-time training monitoring
- 16:27Failure handling and debugging
- 18:36Logging models to registry
- 19:18Running partitioned inference
- 21:04Performance benchmarks
- 22:29Q&A
related talks
28mEfficient Time-Series Forecasting with Thousands of Local Models on Databricks
29mScaling Python-Powered Machine Learning With Snowflake
22mScalable AutoML for Time Series Forecasting
24mTraining large scale models using PyTorch
33mOvercoming Distributed ML Challenges with Ray Train
46mScaling ML Pipelines with Feast, Ray and Kubeflow
22mLarge Scale Data Loading and Data Preprocessing with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery