Combating sampling bias in production How to collect and interpret performance data to drive growth
summary
Production perf metrics suffer three concrete sampling biases: overrepresentation of heavy users (data points ≠ users), underrepresentation of churned or never-adopting users who drop out or never try the app, and missing incomplete operations (survivorship bias). Mitigations are specific: consolidate multiple events per device into a single user-level datum, capture failures to compute success rate including incomplete ops, and validate impact with holdback/AB experiments (example: improving time to first tweet yields measurable DAU growth at scale).
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 3:18Why use production performance metrics
- 7:38Overrepresentation of heavy users
- 14:30Underrepresentation of lost users
- 25:03Missing data from incomplete operations
- 32:52Tying sampling biases to user growth
- 35:03Summary and Q&A
related talks
34mTen Tips to Leverage Data Better and Make Better Products
12mPerformance In Context
51mProduction Profiling: What, Why and How
23mBig data and big insights
50mDesigning for Performance
06mMeasurement is Not Enough: Rise of Performance Analytics
34mThe Business of Performance, a Panel discussion
32mStatistics for Engineers