Replication: How Well Do My Results Generalize Now? The External Validity of Online...
summary
Replication compares MTurk (n=800) and two Prolific samples (gender-balanced n=800, representative n=800) to Pew (n=4272) and finds MTurk raw quality has fallen with 39.1% fail the free‑text attention check, while Prolific samples match Pew on perceptions, experiences and beliefs but not on factual knowledge or behavioral questions. Raking on age/education/sex partly helps; best results come from excluding attention‑check failures and using text responses to catch bots.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:20How Have the Online Platforms That People Use for Surveys Changed since 2017
- 2:06Generalizability
- 3:25Experience Questions
- 3:36Knowledge Questions
- 3:54Perceptions Questions
- 4:09Beliefs Questions
- 5:13Gender Balance Sample
- 5:57Prolific Representative Sample
- 6:41Total Variation Distance
- 7:20What Are the Effects of Data Quality Measures on Generalizability and Based on that What Are Current Best Practices for Online Surveys
- 9:36Summary
related talks
38mNo Traffic, No Users, No Problem! - Usability Testing for New Apps
37mOptimal on Paper, Broken in Reality
01h 16mHow METR measures Long Tasks and Experienced Open Source Dev Productivity
52mAB-tests, lies, damned lies, and statistics.
25mPublishing, Partnership & Problems
07mAre we doing this right?
30mHow to Win Docs and Influence Robots: Improving Robot Accuracy Through Better Docs
25mGotta catch'em all: recognizing sloppy work in crowdsourcing tasks