Alternative Bayes factors: Sample size determination and discriminatory power assessment

Abstract

Alternative Bayes factors are families of methods used for hypothesis testing and model selection when sensitivity to priors is a concern and also when prior information is weak or lacking. This paper deals with two related problems that arise in the practical use of these model choice criteria: sample size determination and evaluation of discriminatory power. We propose a pre-experimental approach to cope with both these issues. Specifically, extending the evidential approach of Royall (J Am Stat Assoc 95(451):760–780, 2000) and following De Santis (J Stat Plan Inference 124(1):121–144, 2004), we propose a criterion for sample size choice based on the predictive probability of observing decisive and correct evidence. The basic idea is to select the minimal sample size that guarantees a sufficiently high pre-experimental probability that an alternative Bayes factor provides strong evidence in favor of the true hypothesis. It is also argued that a predictive analysis is a natural approach to the measurement of discriminatory power of alternative Bayes factors. The necessity of measuring discrimination ability depends on the fact that alternative Bayes factors are, in general, less sensitive to prior specifications than ordinary Bayes factors and that this gain in robustness corresponds to a reduced discriminative power. Finally, implementation of the predictive approach with improper priors is discussed and possible strategies are proposed.