Mathematical evidence for the adequacy of Bayesian optional stopping


The practice of sequentially testing a null hypothesis as data are collected until the null hypothesis is rejected is known as optional stopping. It is well-known that optional stopping is problematic in the context of null hypothesis significance testing: The false positive rates quickly overcome the single test’s significance level. However, the state of affairs under null hypothesis Bayesian testing, where p-values are replaced by Bayes factors, is perhaps surprisingly much less consensual. Rouder (2014) used simulations to defend the use of optional stopping under null hypothesis Bayesian testing. The idea behind these simulations is closely related to the idea of sampling from prior predictive distributions. In this paper we provide formal mathematical derivations for Rouder’s approximate simulation results for the two Bayesian hypothesis tests that he considered. The key idea is to consider the probability distribution of the Bayes factor, which is regarded as being a random variable across repeated sampling. This paper therefore offers a solid mathematical footing to the literature and we believe it is a valid contribution towards understanding the practice of optional stopping in the context of Bayesian hypothesis testing.

Manuscript under review