Evaluating Time-to-Event Models is Hard
summary
Uses tidymodels and yardstick to evaluate censored time‑to‑event models with reverse Kaplan Meier to estimate censoring probabilities and inverse probability of censoring weights as case weights. Runs on a 1,000‑row validation set across evaluation times 5 to ~225, usable cases drop to ~250 by time 200. Reports integrated Brier ≈ 0.07, Brier and ROC AUC as functions of time, shows calibration breaks and noisy estimates around time 200 while ROC AUC still reaches about 0.75.
AI-written summary · about
chapters (12)
- 0:00Evaluating time-to-event models
- 1:03Why evaluation is hard
- 3:57Ambiguous censored data
- 6:24Reverse Kaplan Meier weights
- 8:53Brier scores for calibration
- 10:00Brier scores in tidymodels
- 11:45Brier score trends
- 12:29Calibration plots for survival
- 14:14ROC AUC for discrimination
- 15:41Calibration vs discrimination
- 16:25Conclusion and tidymodels
- 17:06Q&A
related talks
18mCensored - Survival Analysis in Tidymodels- 18mCensored - Survival Analysis in Tidymodels
01h 24mBayesianism and Survival Analysis
25mParadoxes in model training and evaluation under constraints
22mConformal Inference with Tidymodels
08mProbabilistic Analysis of WCETs on RT Systems: When Your Worst Case Isn't Worst
29mEvaluation Design
01h 19mModel criticism with ArviZ