The Dangers of Mindless Forecasting
summary
Warning that maximization of metrics often produces brittle models; high‑AUC ensembles and neural nets can overfit noisy data, while a simple classification tree gives similar real‑world accuracy. Python packages make complex models easy but expensive to train; a recommender prototype on ~3,000 classes using random forest runs slowly and fails to predict for 20–30% of classes. Rule‑of‑thumb claims include 40 lines per variable and small samples amplify spurious correlations; a working prototype exists but simpler models prove more robust.
AI-written summary · about
chapters (18)
- 0:00Mindless forecasting dangers
- 1:16High-metric models fail
- 2:23Models as incomplete representations
- 3:30Simple vs complex modeling
- 5:21Modeling is always a cycle
- 7:27Simpler models superior
- 9:15Titanic survival case study
- 11:42Training vs testing data sets
- 13:33Overfitting dangers
- 14:20Simple vs complex models
- 17:08Complexity vs explainability
- 18:11Forecasting in changing systems
- 19:59Small samples and randomness risks
- 21:30Too many variables pitfalls
- 22:58Estimated vs measured variables
- 24:03Company showcase challenges
- 27:59Book on economic prediction
- 29:29Q&A on correlation
related talks
45mTop 10 Machine Learning Gotchas
39mForecasting With Classical and Machine Learning Methods
17mCodifying Data Science Intuition
42mCreating correct and capable classifiers
40mCART: Not only Classification and Regression Trees
48mPrinciples and Practices of Building Machine Learning Systems
37mCreating correct and capable classifiers
15mMachine Learning: 5 Additional Metrics to Avoid High‑Performing but Useless Models