Intro to ML: How to Prevent Data Leakage and Build Efficient Workflows
summary
Shows exactly how to stop leakage by learning preprocessing only on training folds and reusing those parameters for test and production, comparing a slow manual workflow with a correct scikit-learn Pipeline and ColumnTransformer implementation and a feature-engine option. Reproduces a paper finding leakage in at least 400+ papers across 17 fields. Demonstrates failure modes (get_dummies creating mismatched columns, duplicates, target-derived features), explains fit/transform/predict semantics, saves one serialized pipeline, and runs working Python notebooks on GitHub.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:52Data leakage importance
- 3:20Understanding machine learning pipelines
- 8:03What is data leakage
- 12:40Causes of leakage
- 19:47Preventing data leakage
- 21:58Transformers and pipelines
- 30:37Demo begins
- 1:00:00Manual preprocessing
- 1:14:08Simulating pipelines
- 1:20:40Scikit-learn Transformers
- 1:30:32Column Transformers
- 1:35:49Scikit-learn pipelines
- 1:42:29Complex ML pipelines
- 1:47:09Automated frameworks
related talks
32mReliable Machine Learning
18mAutomate the boring ML stuff with pipelines
27mScalable Data Pipelines - an ML Approach
44mPrivacy by Design Machine Learning
27mA Look Inside Applied Machine Learning
39mDoing Data: The Critical Process of Data Preparation (...)
43mMachine Learning Engineering principles with Python and MLFlow
21mHow The Application Of Machine Learning Is Changing The Way Companies Use Data