Hypothesis Testing Stop Criterion with "Precision Is The Goal"
summary
Claims that Bayesian methods do not magically fix sequential stopping and instead recommends stopping on interval precision is the goal rather than p-value peeking. Demonstrations use Bernoulli coin simulations, compare NHST (peaks causing false rejections ~30–40% when stopping early), HDI + ROPE (still rejects ~20% on true null), and a combined conservative precision is the goal rule that keeps estimates unbiased while avoiding endless inconclusive runs. Slides and a Streamlit sample calculator support reproducing the experiments.
AI-written summary · about
chapters (14)
- 0:00Start
- 1:57The Stopping Criteria
- 3:27Take-Home Messages
- 5:17What Hypothesis Testing Is
- 8:34Coin Toss
- 9:17When Should You Use Frequencies versus Bayesian
- 11:50The One-Sided Test
- 12:38Examples
- 21:50The Stop Criterion
- 22:34Results
- 27:44How Well Does the Model Perform
- 28:32Is Variant a Better than Variant B
- 29:58Working Examples on the Calculator
- 33:38Multi-Objective Optimization
related talks
30mHypothesis Testing Stop Criterion with “Precision Is The Goal”
34mHypothesis to Impact: Mastering the Art and Science of Controlled Experiments
39mSuccessfully Applying Bayesian Statistics to A/B Testing in Your Business
32mHypothesis selection with computational constraints
38mUsable A/B testing – A Bayesian approach
28mData Driven Convergence in Statistical Estimators
47mexpectation: A modern take on statistical A/B testing with e-values and martingales
37mRealtime Bayesian A-B testing with Spark Streaming