Machine Learning Model Evaluation Metrics
summary
Shows concrete comparisons and tools: logistic regression scored 96% accuracy while a dummy classifier scores 94.2% on a 10,000-sample set with class imbalance 5% positive, uses scikit-learn confusion matrices and grid search, demonstrates Titanic model at 83.4% accuracy, explains precision/recall/F1 tradeoffs, flags MCC returning zero on pathological splits, and contrasts ROC AUC 0.92 vs PR 57.7% to illustrate when ROC misleads on imbalanced data.
AI-written summary · about
chapters (26)
- 0:00Evaluation metrics matter
- 1:14What is an evaluation metric
- 2:17Training and evaluating models
- 4:05Classification metrics
- 4:47Accuracy limitations
- 6:35Class imbalance and dummy classifiers
- 8:00Confusion matrix
- 10:55Precision and recall
- 12:46F1 score and business
- 13:53Matthews correlation coefficient
- 15:21MCC versus F1
- 17:58ROC curve and AUC
- 20:49Precision-recall curve
- 21:31Log loss and calibration
- 23:19Multi-class metrics
- 24:24Averaging methods
- 27:15Multi-class log loss
- 28:41Regression metrics
- 29:25R-squared and fit
- 30:32Mean absolute error
- 32:01RMSE sensitivity
- 33:25Logarithmic error
- 34:53Mean absolute percentage error
- 35:36Metrics pitfalls
- 36:59Metrics can be gamed
- 37:20Choosing the right metric
related talks
35mML: Beyond confusion matrix, model performance analysis
01h 33mUnderstanding and diagnosing your machine-learning models
27mUsing Machine Learning to solve a classification problem with scikit-learn
51mHow to Test Artificial Intelligence and Machine Learning Solutions?
57mHow To Evaluate and Improve Machine Learning Models
43mQuality Of Machine Learning Models
42mCreating correct and capable classifiers
27mModel Evaluation for Humans