Beyond the P-Value: A Data Scientist’s Guide to Tier-1 Feature Launches
summary
A Jupyter Notebook walkthrough shows messy launches: 130K users opted in and a headline 38% increase with a tiny p‑value, but naive t tests overstate the effect (4.28 → 1.00 after PSM). Methods include propensity score matching with sklearn or PyMatch, uplift modeling, and spillover detection (each treated teammate adds about 0.4 videos/month to controls). Code uses SciPy, CausalML and causal uplift learners, team-level randomization, and standardized mean differences to prove whether the reported lift is real.
AI-written summary · about
chapters (19)
- 0:00Data science pitfalls
- 1:09Launch postmortem example
- 2:13Standard metrics mislead launches
- 3:18Loom editing feature impact
- 5:46Critical assumptions to question
- 7:08Naive baseline analysis with T-tests
- 7:50Selection bias in treatment groups
- 8:52Spillover effects and SUTVA violations
- 10:17Correcting spillover bias
- 12:01Opt-in trap and power user bias
- 13:05Propensity score matching step-by-step
- 15:35Validating matching results
- 16:17Standardized mean difference checks
- 17:39True causal impact beyond means
- 18:43Heterogeneous effects and segments
- 20:09Predicting uplift for user groups
- 21:51Targeting high-value users
- 22:12Key takeaways for launches
- 23:57Q&A
related talks
38mHow to Enhance Customer Targeting in Marketing
19mUnveiling the Potential: Exploring Upselling, pLTV, and ML for Operations
43mBeyond Just Prediction: Causal Thinking in Machine Learning
34mPropensity Score Matching: A Non-experimental Approach to Causal...
32mWhy start using uplift models for more efficient marketing campaigns
11mFrom 0 to 10,000 experiments
39mExperimentation and the gold standard of data champions
01h 15mAB-testing by cluster