~/talk

Experimentation at Stack Overflow: How we built our new comments UI

Talkat Agile on the Beach 2026 · 2026-07-02..07-03· [data-engineering]
slides:none on file

Stack Overflow replaces a fragile legacy A/B system by reusing feature flags, eventing and Databricks, implementing a salted hash per feature to deterministically split traffic and track events. A live nine-variant ABCDEFGHI test reveals practical pitfalls: an opt-out bias problem that skews group composition, a US holiday confounder, and a coding mistake that leaked results. Measured outcomes: showing a profile picture increases upvotes, smaller font raises comment frequency but lowers votes, and large multichange UI swaps hide which variable matters.

AI-written summary · about

  1. 0:00Introduction to experimentation
  2. 1:09What is agile experimentation
  3. 4:59Four ways to get user feedback
  4. 6:03Building AB testing from existing tools
  5. 11:00Splitting traffic with feature flags
  6. 13:31Tracking user behavior with events
  7. 14:32Analyzing data with AI-assisted tools
  8. 16:15Story of building new comments UI
  9. 17:18First experiment with comment dropdown
  10. 21:51Opt-out feature and data analysis issues
  11. 24:40Full UI redesign and Meta feedback
  12. 27:10Single-variable experiments for clarity
  13. 29:58Multi-variable AB testing approach
  14. 32:03Running eight UI variants simultaneously
  15. 33:28Results and data interpretation
  16. 40:27Impact on comment quality and engagement
  17. 41:28Summary and closing remarks
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.