On the Use of ML for Blackbox System Performance Prediction
summary
Evaluates six ML models on 13 real applications and finds 12 of 13 show inherent variability that limits prediction accuracy, with the oracle error exceeding 15% for five apps under best‑case assumptions. System fixes (disable optimizations, remove bimodal behaviors) cut oracle error to under 10% for 90% of cases, but this requires invasive changes and tradeoffs. Probabilistic models such as mixture density networks and probabilistic random forests improve results yet prediction error remains high in many realistic scenarios, so ML is neither simple nor broadly general.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:25Performance prediction is increasingly important! • Optimization, capacity planning, SLO-aware scheduling
- 0:52Challenges
- 1:33ML for system perf. prediction?
- 3:47Applications and Models
- 4:34Metrics and Predictors
- 5:24Best Case Test Results
- 7:04E.g., Spark worker readiness
- 7:48With system modifications
- 8:49Embrace variability: probabilistic predictions
- 10:23What if we go beyond the best case ?
- 10:53Methodology Blueprint
- 11:16Conclusion
related talks
22mPredict the future (or don't) with machine learning in applications
22mHow to Detect Silent Failures in ML Models
25mWell well ML is not the answer to everything- 22mHow to Detect Silent Failures in ML Models
39mMethodology of ML in Performance Testing (How to Find Bottlenecks Automatically)
35mOn Machine Learning for Prediction and Prioritization in the Allocation of Scarce Societal Resources
09mML and AI systems will make mistakes
39mForecasting With Classical and Machine Learning Methods