Experimentation at Stack Overflow: How we built our new comments UI
summary
Stack Overflow replaces a fragile legacy A/B system by reusing feature flags, eventing and Databricks, implementing a salted hash per feature to deterministically split traffic and track events. A live nine-variant ABCDEFGHI test reveals practical pitfalls: an opt-out bias problem that skews group composition, a US holiday confounder, and a coding mistake that leaked results. Measured outcomes: showing a profile picture increases upvotes, smaller font raises comment frequency but lowers votes, and large multichange UI swaps hide which variable matters.
AI-written summary · about
chapters (17)
- 0:00Introduction to experimentation
- 1:09What is agile experimentation
- 4:59Four ways to get user feedback
- 6:03Building AB testing from existing tools
- 11:00Splitting traffic with feature flags
- 13:31Tracking user behavior with events
- 14:32Analyzing data with AI-assisted tools
- 16:15Story of building new comments UI
- 17:18First experiment with comment dropdown
- 21:51Opt-out feature and data analysis issues
- 24:40Full UI redesign and Meta feedback
- 27:10Single-variable experiments for clarity
- 29:58Multi-variable AB testing approach
- 32:03Running eight UI variants simultaneously
- 33:28Results and data interpretation
- 40:27Impact on comment quality and engagement
- 41:28Summary and closing remarks
