AB-tests, lies, damned lies, and statistics.
summary
A newsroom redesign that took months shows no user change, while a tiny UI tweak — red play button gives +5% views — moves metrics, proving effort does not equal impact. Implementation uses bucket ID cookie distribution (1–100 buckets, persistent but rotating) and Google Analytics custom dimensions. Statistical points: large sample needed for 1% effects (millions for 99% confidence). Tests include a negative experiment where random recommendations don’t help, and validation uses proxy tests with BrowserMob/Selenium to catch tracking bugs.
AI-written summary · about
chapters (44)
- 0:00Introduction
- 3:41Who am I
- 5:20ABtestdriven development
- 5:52Doing the right thing
- 6:32Biases
- 7:17ABtest
- 8:38Randomness
- 11:07Questioning
- 11:37Making better decisions
- 13:08KPIs
- 14:00Page Views
- 14:31User Testing
- 17:27Optimizely
- 18:14Statistics
- 28:23User Journey
- 30:20Learning to make better decisions
- 30:59The naive approach
- 32:39Dividing concrete
- 32:57Buckets
- 34:01Bucket ID
- 34:36Bucket Cookie
- 36:18Bucket IDs
- 37:21Google Analytics
- 38:04Dashboards
- 39:05histogram
- 39:29tracking all tests
- 39:42red button test
- 40:01negative testing
- 42:34nobody uses the data
- 43:05focus on the wrong metrics
- 44:47sampling
- 45:25significance metrics
- 45:59broken code
- 46:13proxy testing
- 46:39browser mode proxy
- 47:34selenium
- 48:25curve for change
- 48:46double verification
- 49:23subjectivity
- 50:18quantify everything
- 50:38be context aware
- 50:59feel your curiosity
- 51:16testdriven development
- 51:29express development
related talks
45mDon't trust the numbers!
36mA/B Testing in Production
44mA B Testing: Harder than just a color change
28mMeasurement and Analysis for Sustainable Service Delivery
56mEssential Analytics and Coding for the ‘Complete’ UX Professional
28mOptimizing your A/B testing roadmap
27mCustomer Driven Development : What, Why & How?
41mA/B Testing with Style