Hindsight bias: A machine learning story
summary
Salesforce Einstein builds an automated ML pipeline that detects and removes label leakage and related dirty-data problems at scale, using Pearson correlation, Kramer's V and decision-tree bucketing to flag overly predictive buckets, plus JS divergence to detect drift and cardinality/type checks to find disguised categorical/text fields. Metadata tracks feature lineage so problematic derivations are dropped. On hundreds of customers the auto-ML removes more leakage than manual tuning and yields much steadier live AUPR/AUROC versus baseline models that collapse in production.
AI-written summary · about
chapters (10)
- 0:00Introduction and welcome
- 1:15Enterprise vs consumer ML
- 5:12Hindsight bias examples
- 8:25Titanic data leakage
- 12:20Data bottlenecks in ML
- 13:25Automated ML at Salesforce
- 17:57AutoML vs manual cleaning
- 19:04Live prediction results
- 21:50Key takeaways
- 22:52Q&A session
related talks
32mReliable Machine Learning
47mMetrics Driven Machine Learning Development at Salesforce Einstein
37mMachine Learning, The Right Way
27mScalable Data Pipelines - an ML Approach
38mFantastic ML apps and how to build them
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era
30mOperationalizing Machine Learning
08mLeveraging data insights to increase efficiency and innovation