Using Simpson’s Paradox to Discover Interesting Patterns in...
summary
Package claims it detects Simpson’s paradox automatically in large behavioral datasets by fitting a global logistic/linear trend then disaggregating on candidate covariates, using decision-tree binning for high‑cardinality features and per‑bin regressions. Implementation mixes C++ (speed) and Python (change analysis, plotting), writes CSVs and PDFs, reports R-squared improvement and significance, finds concrete reversals (StackExchange acceptance vs session length, gambling vs repayment when conditioning on time), and currently supports binary outcome only.
AI-written summary · about
chapters (12)
- 0:00Simpson’s Paradox intro
- 1:11Misleading trends examples
- 2:36What is Simpson’s Paradox
- 3:40Case study: Smokers and mortality
- 5:43Simpson’s Paradox in credit scoring
- 6:46Algorithms for detection
- 9:56Handling confounding variables
- 11:42Real dataset applications
- 13:31Live detection demo
- 17:16Algorithm on sample data
- 21:54Summary and future work
- 22:40Q&A
related talks
26mParadoxes in Data Science
31mLogistic Regression Behind the Scenes
33mHidden Biases In Data: Moving Beyond Black Box Models
15mInterpretable Machine Learning with R
26mDiscovering Unwarranted Associations in Data-Driven...
15mThe tidysynthesis R package
47mThe Elephant in your Dataset Addressing Bias in Machine Learning
38mMixed and Covariate-Dependent Graphical Models