Supercharging Pipeline Efficiency with ML Performance Prediction
summary
Singular builds a pipeline that predicts task runtime and memory to split hundreds of thousands of daily Celery tasks into light and heavy queues (>90% light, ~10% heavy). It trains per-customer SVR models daily on S3/Athena data, serializes models to Redis, and has the Celery router score each task at enqueue time. Kubernetes HPA uses a moving-rollup metric from Datadog, plus VPA and cluster autoscaler, to scale short-task workers on AWS Spot instances, eliminating queue snowballing and OOM kills.
AI-written summary · about
chapters (8)
- 0:00Introduction to pipeline challenges
- 2:56Problem of task duration imbalance
- 4:21Solution with ML performance prediction
- 9:38Productionizing the ML model
- 13:06Auto scaling with Kubernetes
- 17:18Cost optimization with spot instances
- 18:01Summary and future directions
- 19:29Q&A session
related talks
25mSupercharging Pipeline Efficiency with ML Performance Prediction
32mMaking Happy Modelers: Build and Maintain...
38mReal Time ML Pipelines in Multi-Tenant Environments
14mPerformance Optimization in High-Volume Transaction Systems
46mScaling ML Pipelines with Feast, Ray and Kubeflow
14mSupercharging ML Pipelines with Go
54mScaling Machine Learning Pipelines in Cloud
24mTaming Data Skew in Production ML Pipelines