Pandas, Pipelines, and Custom Transformers
summary
Shows that scikit-learn Pipelines strip DataFrame labels and offers a practical fix: implement pandas-friendly Pipeline and FeatureUnion and a set of custom transformers (column extractor, wrapper for StandardScaler, dummy transformer, log1p filler). Provides code on GitHub, applies to Seattle permits data, and reports AUC improving from 0.58 to 0.71 after the feature pipeline. Demonstrates how unseen categorical levels are handled and how transformers keep index and column names.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:11Simple ML model
- 6:33Scikit-learn transformers
- 9:03Pipelines for transformations
- 11:37Categorical data with one-hot
- 13:45Custom transformers
- 17:44Pandas and scikit-learn issues
- 20:13Complex feature pipeline
- 21:42Model performance evaluation
- 22:28Q&A
related talks
43mDeploying Machine Learning using sklearn pipelines
33mUsing Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software
36mBuild prod ML systems with feature /training /inference pipelines
28mIdiomatic Pandas
29mGetting Scikit-Learn To Run On Top Of Pandas
01h 31mPandas, Data Wrangling & Data Science
30mTesting Pandas- Shoots, leaves, and garbage!
21mHow to use Pandas Efficiently