Supercharging Pipeline Efficiency with ML Performance Prediction
summary
A productionized pipeline-classifier predicts task duration and memory per job (serialized per-customer SVR models stored in Redis and retrained daily), Celery’s router uses those predictions to send tasks to dedicated light or heavy queues, Kubernetes horizontal/vertical autoscaling spins workers by queue length and runs short workers on AWS Spot instances, 90% of tasks prove to be short so queueing drops and cost falls, predictions use RMSE monitoring to trigger investigation, the system ships and measurably reduces delays and OOM kills.
AI-written summary · about
chapters (9)
- 0:00Start
- 3:03What's the problem?
- 4:56Classifying tasks based on input parameters?
- 9:27Choosing our Features: Account
- 13:59Model Performance
- 15:07Why Bother with Autoscaling?
- 18:57Cost Optimization
- 19:34Summary
- 20:13What's Next?
related talks
25mSupercharging Pipeline Efficiency with ML Performance Prediction
14mPerformance Optimization in High-Volume Transaction Systems
25mContinuous ML Applications in Production
25mThe Hidden Cost of ML Data Lifecycles in Kubernetes
37mAutomated Machine Learning: Building a Pipeline
32mMaking Happy Modelers: Build and Maintain...
12mUse ML to Predict Delivery Time and Improve Code Reliability
24mTaming Data Skew in Production ML Pipelines