The Forecast Whisperer: Secrets of Model Tuning Revealed
summary
Compares MAE, MSE and WMAPE on sparse retail series and shows MAE training produces a -6,000 profit while switching loss to MSE yields +3,000 and tuning for monthly WMAPE (via Optuna) doubles WMAPE and increases profit further. Fixes model architecture (a Temporal Fusion Transformer example) and iterates on data: detect change points, drop long zero-run supply-shortage periods, and clean mislabeled promotions flagged by Shapley values. Demonstrates business-objective tuning (including directly optimizing profit as tuning objective) produces the highest real profit.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:11Business objectives for models
- 4:44Metrics versus business outcomes
- 6:51Loss functions for sparse data
- 8:39Tuning models for profit
- 11:07From model-centric to data-centric
- 14:14Supply shortages in time series
- 16:23Explainability for data issues
- 19:52Key takeaways
- 21:39Q&A
related talks
35mWhat Failure Taught Me About Building High-Stakes Models
30mMy forecast is better than yours! What does that even mean?
17mCodifying Data Science Intuition
33mTune your ML models to the highest accuracy with automatic model tuning
19mMinimize Risks and Accelerate MLOps with Model Performance Monitoring and Explainability
23mTotal Tidy Tuning Techniques
31mML time series forecasting methods
29mProbabilistic demand forecasting at scale