Did It Work? Measuring the Impact of Product Launches
summary
Shows how to answer "did it work" with concrete methods: use A/B tests when feasible but expect setup time and partial rollout tradeoffs; compare users who adopt the feature versus those who don't; align metrics to zero-based time alignment (days since first use) to reveal delayed effects; and remove selection bias with propensity matching for causal estimates. Recommends using event-based analytics (Mixpanel-style) to run these segmentations and claims combining these three techniques lets teams credibly say whether a launch moved the metric.
AI-written summary · about
chapters (8)
- 0:00Intro
- 1:00Mikes background
- 4:31What happened
- 5:00Whats happening
- 7:00Trends
- 7:40AB Testing
- 8:53AB Testing Results
- 10:11Conclusion
related talks
28mMeasurement and Analysis for Sustainable Service Delivery
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
13mData for product
32mHow to evaluate team effectiveness with a data first approach
30mMeasurement in a Privacy First World
23mHow Do You Get More of Product Insight
40mReal-time A/B: how to get metrics for experiments
14mBuilding a Data-Driven Culture: An Anthology of Strategies and Insights