Investigating the relationship between the Bayes factor and the separation of credible intervals

Abstract

We examined the relationship between the Bayes factor and the separation of credible intervals in between- and within-subject designs under a range of effect and sample sizes. For the within-subject case, we considered five intervals: (1) the within-subject confidence interval of Loftus and Masson (1994); (2) the within-subject Bayesian interval developed by Nathoo et al. (2018), whose derivation conditions on estimated random effects; (3) and (4) two modifications of (2) based on a proposal by Heck (2019) to allow for shrinkage and account for uncertainty in the estimation of random effects; and (5) the standard Bayesian highest-density interval. We derived and observed through simulations a clear and consistent relationship between the Bayes factor and the separation of credible intervals. Remarkably, for a given sample size, this relationship is described well by a simple quadratic exponential curve and is most precise in case (4). In contrast, interval (5) is relatively wide due to between-subjects variability and is likely to obscure effects when used in within-subject designs, rendering its relationship with the Bayes factor unclear in that case. We discuss how the separation percentage of (4), combined with knowledge of the sample size, could provide evidence in support of either a null or an alternative hypothesis. We also present a case study with example data and provide an R package ‘rmBayes’ to enable computation of each of the within-subject credible intervals investigated here using a number of possible prior distributions.

Appendices

Appendix A

Proof of Theorem 1

After some substitutions, the Pearson Bayes factor in Equation 9 becomes

$$\text{P-BF}_{10}=\frac{\Gamma \left(\frac{a+1}{2}+\upgamma \right)\cdot \Gamma \left(\frac{a\left(n-1\right)}{2}\right)}{\Gamma \left(\frac{an-1}{2}\right)\cdot \Gamma \left(1+\upgamma \right)}{\left(1+\frac{{{SS}}_{\mathrm{B}}}{{{SS}}_{\mathrm{W}}}\ \right)}^{\frac{a\left(n-1\right)}{2}-1-\upgamma}.$$

(A1)

By applying Stirling’s formula Γ(y + z) ∼ y^zΓ(y) as y →  + ∞, the gamma ratio in Equation A1 becomes $\frac{\Gamma \left(\frac{a\left(n-1\right)}{2}\right)}{\Gamma \left(\frac{an-1}{2}\right)}\sim {\left(\frac{an}{2}\right)}^{\frac{1-a}{2}}$ as n →  + ∞.

We calculated the separation for the standard between-subjects confidence interval in Equation A2 as the absolute value of the difference between two sample means over the twofold interval width. Here, we consider only a = 2.

$$Sep=\left|{M}_{1\cdot }-{M}_{2\cdot}\right|/\left(2{t}_{1-\frac{\alpha }{2},\ a\left(n-1\right)}^{\ast}\sqrt{\frac{{{SS}}_{\mathrm{W}}}{n\left(n-1\right)a}}\right).$$

(A2)

As n →  + ∞, ${t}_{1-\frac{\alpha }{2},\ a\left(n-1\right)}^{\ast}\sim {z}_{1-\frac{\alpha }{2}}$, the sample means converge to the population means, and (SS_B/SS_W)² is assumed to be the least significant in the exponential series. ${SS}_{\mathrm{B}}=n\overset{a}{\sum\limits_{i=1}}{\left({M}_{i\cdot }-M\right)}^2$ reduces to $\frac{1}{2}n{\left({M}_{1\cdot }-{M}_{2\cdot}\right)}^2$when a = 2. Plugging Equation A2 in the limit below, we obtain

$${\displaystyle \begin{array}{ll} & \underset{n\to +\infty }{\lim}\left[{\left(1+\frac{SS_{\mathrm{B}}}{SS_{\mathrm{W}}}\right)}^{\frac{a\left(n-1\right)}{2}-1-\upgamma}-\exp \left\{{z}_{1-\frac{\upalpha}{2}}^2\cdot {Sep}^2\right\}\right]\\ {}=& \underset{n\to +\infty }{\lim}\left[{\left(1+\frac{SS_{\mathrm{B}}}{SS_{\mathrm{W}}}\right)}^{\frac{a\left(n-1\right)}{2}-1-\upgamma}-\exp \left\{\frac{1}{2}a\left(n-1\right)\frac{SS_{\mathrm{B}}}{SS_{\mathrm{W}}}\right\}\right]\\ {}=& \underset{n\to +\infty }{\lim}\left[{\left(1+\frac{SS_{\mathrm{B}}}{SS_{\mathrm{W}}}\right)}^{\frac{a\left(n-1\right)}{2}-1-\upgamma}-{\left(1+\frac{SS_{\mathrm{B}}}{SS_{\mathrm{W}}}+O\left({\left(\frac{SS_{\mathrm{B}}}{SS_{\mathrm{W}}}\right)}^2\right)\right)}^{\frac{a\left(n-1\right)}{2}}\right]=0.\end{array}}$$

Hence, the asymptotic approximation of the log Pearson Bayes factor is a quadratic function of the separation of the standard confidence interval for population means in balanced one-way between-subjects designs, as the number of subjects goes to infinity. To check the convergence of the limit, we plot the (solid black) fitted line for the relationship between the log JZS-BF₁₀ and squared separation score, along with the (dashed red) analytic line from plugging the known values into the formula of Theorem 1, in Fig. 10. As the sample size increases, the two lines become closer. We still expect some variations between these lines even for very large n because Theorem 1 applies for the Pearson Bayes factor, whereas the quadratic exponential is interpolated for the JZS Bayes factor.

Fig. 10

Plots of the relationship between the log Bayes factor and the squared separation score of the standard confidence interval for population means in between-subjects designs, with the fitted line in solid black and the analytic line in dashed red

Appendix B

R packages to perform Bayesian inference

The ‘rmBayes’ package performs Bayesian interval estimation for both the homoscedastic and heteroscedastic cases in either between- or within-subject designs that include a single independent variable. The Stan-based R source package installation will take a few minutes because models need to be compiled into dynamic shared objects. We recommend using R version 4.0.1 or later and installing the pre-compiled binary package so users do not have to worry about C++ compiler issues. The relevant commands are:

> install.packages("rmBayes", type = "binary")

> library(rmBayes)

The rmHDI function in ‘rmBayes’ provides multiple methods to construct the credible intervals for population means, with each method based on different sets of priors. The default method implements the NUTS algorithm and constructs the within-subject HDI corresponding to the JZS-HDI case in Table 1. More methods documentation can be viewed on GitHub, https://zhengxiaouvic.github.io/rmBayes/. The following example includes a partial data set, a call to the rmHDI function, and the resulting output. The partial data set is also shown in wide format.

> ## Data are in the long format. 10 subjects. 3 conditions.

> head(recall.long, 2)

Subject Level Response

1 s1 Level1 10

2 s2 Level1 6

> rmHDI(recall.long, whichSubject = "Subject", whichLevel = "Level", whichResponse = "Response", seed = 277) #macOS (Apple chip)

$HDI

lower upper

Level1 10.47101 11.59361

Level2 12.39176 13.51436

Level3 13.55086 14.67346

$`posterior means`

Level1 Level2 Level3

11.03231 12.95306 14.11216

$width

[1] 0.5613014

> ## Same data are in the wide format.

> head(recall.wide, 2)

Level1 Level2 Level3

s1 10 13 13

s2 6 8 8

> rmHDI(data.wide= recall.wide, seed = 277)

An alternative method for computing HDIs is possible using the ‘BayesFactor’ package, which computes Bayes factors for several experimental designs. The anovaBF function can first be used to generate the Bayes factor for a within-subject design.

> library(BayesFactor); set.seed(277)

> anovaBF(Response ~ Level + Subject, data = recall.long, whichRandom = "Subject", iterations = 100000, progress = FALSE)

Bayes factor analysis

--------------

[1] Level + Subject : 36469.12 ±0.32%

Against denominator:

Response ~ Subject

---

Bayes factor type: BFlinearModel, JZS

Then, Gibbs sampling can obtain parameter estimates from the posterior distribution of the Bayes factor object numerator. Those estimates are plugged into the interval equations for LH- or JZS-HDI in Table 1 to construct the within-subject HDI. This method can be implemented using the R code below, which defines a new anovaHDI function. Both anovaHDI and the default rmHDI functions assume all the same priors but use different sampling algorithms for establishing posterior distributions.

> anovaHDI <- function(data, whichSubject, whichLevel, whichResponse, cred, iter) {

#' input arguments are defined as in the rmHDI function

n <- length(unique(data[,whichSubject]))

a <- length(unique(data[,whichLevel]))

BF <- BayesFactor::anovaBF(as.formula(paste(whichResponse, "~", whichLevel, "+", whichSubject)), data = data, whichRandom = whichSubject, iterations = iter, progress = FALSE)

chains <- BayesFactor::posterior(BF, iterations = iter, progress = FALSE)

mu.chains <- chains[,2:(a+1)] + chains[,1]

widths <- qt((1 + cred) / 2, df = a * (n - 1)) * sqrt(chains[,"sig2"] / n)

uprs <- mu.chains + widths

lwrs <- mu.chains - widths

matrix(c(colMeans(lwrs),colMeans(uprs)), nrow = a, dimnames = list(paste("Level",1:a), c("lower","upper")))

}

> set.seed(277)

> anovaHDI(recall.long, "Subject", "Level", "Response", .95, 100000)

lower upper

Level 1 10.50752 11.64187

Level 2 12.41498 13.54934

Level 3 13.56208 14.69644

Appendix C

Monte Carlo error and a data permutation issue

Users should expect different results if they vary the number of iterations or the random seed used in MCMC. Such variability is referred to as Monte Carlo error. We examined Monte Carlo error in computing Bayes factors by applying the anovaBF function 500 times (each containing 100,000 MCMC iterations; the default value is 10,000) with different random seeds on the same set of simulated within-subject data. Among these 500 runs, one Bayes factor outlier was as extreme as 11.7, although the vast majority of values ranged from 4.3 to 5.4. R scripts for this and the following examples are available at https://osf.io/x2pvw/.

Similarly, Monte Carlo error is associated with the Bayesian interval estimation. We replicated rmHDI and anovaBF functions 100 times (each containing 20,000 MCMC draws – 2 chains with 10,000 iterations each in rmHDI) with different random seeds on the same simulated within-subject data. Furthermore, we visualized the resulting variability of the posterior mean estimates, HDI widths, and HDI separation via density plots. The whole process was repeated several times for data sets having different Bayes factors, correlations between conditions, and sample sizes. One example is exhibited in Fig. 11. In different realizations of the draws from the posterior distribution, it is also worthwhile to note a data permutation issue that affects the simulation. That is, the same experimental data are used but permuted by row (e.g., switch Subject 10 up to the second place) or by column (e.g., Level-high and Level-low rather than Level-low and Level-high order). Permutation of the entries in a data file will result in slightly different estimates even if the random seed stays the same. We randomly permuted the data by row but fixed the same random seed when calling rmHDI to assess the magnitude of the permutation issue relative to Monte Carlo error. In Fig. 11, two density plots (permuted data but using a constant random seed, or not permuted but varying the random seed) generated from results provided by the rmHDI function in the ‘rmBayes’ package highly overlap, indicating that permutation of the data produces variability in outcomes of a similar magnitude to setting different random seeds. Moreover, the functions in the ‘BayesFactor’ package returned less variability in estimates for posterior means but more variability in estimates for standard error of the mean (thus, interval width) and posterior mean difference, whereas the rmHDI performance is quite the opposite. The separation percentage is less variable when calling rmHDI, as shown in panel E of Fig. 11. Although the models and priors assumed by the two packages are identical, there may be differences in the actual code implementation, especially for Equations 2 and 4 and MCMC samplers (Gibbs sampling in the anovaBF and NUTS in the rmHDI), leading to differences in the variable results.

Fig. 11

Density plots of the simulations from replicating R functions. A: posterior mean difference; B: HDI width; C: posterior mean of one condition; D: posterior mean of the other condition; E: HDI separation percentage. The same random seed was fixed when investigating the permutation issue, denoted as ‘rmBayes (permuted)’, whereas different random seeds were used for investigating the Monte Carlo error in ‘BayesFactor’ and ‘rmBayes’

In the rmHDI function, the default setting for the argument permuted is TRUE, meaning the converted wide-format data are first ordered by their column names in alphabetic order. Then, the data are placed in ascending order by the first and second columns.

Appendix D

Warning messages regarding sampling and effective Monte Carlo sample size

The Stan website https://mc-stan.org/misc/warnings lists all the potential warnings in running an MCMC. Three common warnings are related to the exceeded maximum tree depth (a concern for long execution time), low bulk effective samples size (ESS, indicating posterior means and medians may be unreliable), and low tail ESS (indicating posterior variances and tail quantiles may be unreliable). The relevance of these warnings depends on the specific data being analyzed. Visit the website https://osf.io/x2pvw/ for an example.

We suspect that a high correlation between conditions in a within-subject design might result in slower, inefficient sampling due to a computed likelihood with elongated elliptical contours. The latter two warnings indicate that the sampler is moving slowly. After accounting for the correlation across successive draws of the Markov chain sampler, the ESS is low. For example, if the lag-1 autocorrelation of the MCMC sampling output is high (e.g., above .97), then 2,000 iterations can be worth, say fewer than 100 independent draws. The warning disappears with 10,000 iterations because the effective sample size may then be sufficiently high to cross the threshold in Stan (it might be an ESS of approximately 500).