Everything That You Wanted To Know About P-Values But Were Afraid To Ask
summary
Claim: p-values have no acceptance signal. Evidence: simulated thousands of binomial trials with 256 tosses per experiment and binned p-values, showing a flat p-value distribution under true null so low and high p’s are equally likely. Example: 8/10 gives p≈0.11, scaling to 24/30 drops p to ≈0.014. For small effects (0.52) even thousands of tosses only yield ~80% rejection. Alternatives: Bayesian posterior or preplanned precision/equivalence testing.
AI-written summary · about
chapters (9)
- 0:00P-values introduction
- 1:09What p-values measure
- 3:17P-values reject null only
- 4:40Simulation of p-value distributions
- 6:47P-values and fair coins
- 7:29P-values limitations
- 8:52Bayesian hypothesis acceptance
- 10:17Precision in hypothesis testing
- 10:37Summary
related talks
24mBeyond the P-Value: A Data Scientist’s Guide to Tier-1 Feature Launches
01h 15mAB-testing by cluster
29mIt might look normal but this distribution will ruin your stats
39mExperimentation and the gold standard of data champions
09mEstimating the Value of Pi with a Dartboard and (Not so Much) Luck!
01h 22mDoing frequentist statistics with Scipy
39mThe Duct Tape of Heroes: Bayes Rule
30mThe Value of Null Results