Increasing the replicability for linear models via adaptive significance levels

Abstract

We put forward an adaptive $\alpha$ (type I error) that decreases as the information grows for hypothesis tests comparing nested linear models. A less elaborate adaptation was presented in Pérez and Pericchi (Stat Probab Lett 85:20–24, 2014) for general i.i.d. models. The calibration proposed in this paper may be interpreted as a Bayes–non-Bayes compromise, of a simple translation of a Bayes factor on frequentist terms that leads to statistical consistency, and most importantly, it is a step toward statistics that promotes replicable scientific findings.

Appendices

Appendix 1 The likelihood ratio

Define

$$\begin{aligned} r({\mathbf {y}}|({\mathbf {X}}_i,{\mathbf {X}}_j))=\dfrac{f({\mathbf {y}}|{\mathbf {X}}_i\widehat{\varvec{\delta }}_i,S_i^2{\mathbf {I}}_n)}{f({\mathbf {y}}|{\mathbf {X}}_j\widehat{\varvec{\beta }}_j,S_j^2{\mathbf {I}}_n)} \end{aligned}$$

we will perform the calculations for the hypothesis test

$$\begin{aligned} H_0:\text {Model} M_i versus H_1:\text {Model} M_j. \end{aligned}$$

Indeed, for model $M_i$

$$\begin{aligned} L({\mathbf {y}}|{\mathbf {X}}_i,\sigma _i^2,\varvec{\delta }_i)=\frac{1}{(2\pi )^{n/2}(\sigma _i^2)^{n/2}}\exp \left\{ -\frac{1}{2\sigma _i^2}({\mathbf {y}}-{\mathbf {X}}_i\varvec{\delta }_i)^t({\mathbf {y}}-{\mathbf {X}}_i\varvec{\delta }_i)\right\} . \end{aligned}$$

Since the MLE of $\varvec{\delta }_i$ is $\widehat{\varvec{\delta }}_i=({\mathbf {X}}_i^t{\mathbf {X}}_i)^{-1}{\mathbf {X}}_i^t{\mathbf {y}}$ and the MLE of $\sigma _i^2$ is $S_{i}^2=\dfrac{{\mathbf {y}}^t({\mathbf {I}}-{\mathbf {H}}_i){\mathbf {y}}}{n}$, where ${\mathbf {H}}_i={\mathbf {X}}_i({\mathbf {X}}_i^t{\mathbf {X}}_i)^{-1}{\mathbf {X}}_i^t$

$$\begin{aligned} \sup _{\Omega _0}L({\mathbf {y}}|{\mathbf {X}}_i,\sigma _i^2,\varvec{\delta }_i)=\frac{1}{(2\pi )^{n/2}(S_{i}^2)^{n/2}}\exp \left\{ -\frac{n}{2}\right\} . \end{aligned}$$

For model $M_j$

$$\begin{aligned} L({\mathbf {y}}|{\mathbf {X}}_j,\sigma _j^2,\varvec{\beta }_j)=\frac{1}{(2\pi )^{n/2}(\sigma _j^2)^{n/2}}\exp \left\{ -\frac{1}{2\sigma _j^2}({\mathbf {y}}-{\mathbf {X}}_j\varvec{\beta }_j)^t({\mathbf {y}}-{\mathbf {X}}_j\varvec{\beta }_j)\right\} . \end{aligned}$$

Since MLE of $\varvec{\beta }_j$ is $\widehat{\varvec{\beta }}_j=({\mathbf {X}}_j^t{\mathbf {X}}_j)^{-1}{\mathbf {X}}_j^t{\mathbf {y}}$ and the MLE of $\sigma _j^2$ is $S_{j}^2=\dfrac{{\mathbf {y}}^t({\mathbf {I}}-{\mathbf {H}}_j){\mathbf {y}}}{n}$

$$\begin{aligned} \sup _{\Omega }L({\mathbf {y}}|{\mathbf {X}}_j,\sigma _j^2,\varvec{\beta }_j)=\frac{1}{(2\pi )^{n/2}(S_{j}^2)^{n/2}}\exp \left\{ -\frac{n}{2}\right\} . \end{aligned}$$

Thus, the likelihood ratio is

$$\begin{aligned} r({\mathbf {y}}|({\mathbf {X}}_i,{\mathbf {X}}_j))=\dfrac{\sup _{\Omega _0}L({\mathbf {y}}|{\mathbf {X}}_i,\sigma _i^2,\varvec{\alpha }_i)}{\sup _{\Omega }L({\mathbf {y}}|{\mathbf {X}}_j,\sigma _j^2,\varvec{\beta }_j)}=\left( \frac{S_{j}^2}{S_{i}^2}\right) ^{\frac{n}{2}}=\left( \dfrac{{\mathbf {y}}^t({\mathbf {I}}-{\mathbf {H}}_j){\mathbf {y}}}{{\mathbf {y}}^t({\mathbf {I}}-{\mathbf {H}}_i){\mathbf {y}}}\right) ^{\frac{n}{2}}. \end{aligned}$$

Appendix 2 An expression for b in (10)

Consider linear regression model $M_j: y_v=\beta _1+\beta _2x_{v2}+\cdots +\beta _j x_{vj}+\epsilon _v$ with $1\le v\le n$ and $2\le j\le k$, then

$$\begin{aligned} {\mathbf {X}}_j=\begin{bmatrix} 1&{} x_{12}-{\bar{x}}_2&{}\cdots &{}x_{1j}-{\bar{x}}_j\\ 1&{} x_{22}-{\bar{x}}_2&{}\cdots &{}x_{2j}-{\bar{x}}_j\\ \vdots &{}\vdots &{}\vdots &{}\vdots \\ 1&{}x_{n2}-{\bar{x}}_2&{}\cdots &{}x_{nj}-{\bar{x}}_j\\ \end{bmatrix} \end{aligned}$$

and

$$\begin{aligned} {\mathbf {X}}_j^t{\mathbf {X}}_j=\begin{bmatrix} n&{} 0&{}0&{}\cdots &{}0\\ 0&{} (n-1)s_2^2&{}(n-1)s_2s_3\rho _{23}&{}\cdots &{}(n-1)s_2s_j\rho _{2j}\\ \vdots &{}\vdots &{}\vdots &{}\vdots &{}\vdots \\ 0&{}(n-1)s_2s_j\rho _{2j}&{}(n-1)s_3s_j\rho _{2j}&{}\cdots &{}(n-1)s_j^2\\ \end{bmatrix} \end{aligned}$$

then

$$\begin{aligned} |{\mathbf {X}}_j^t{\mathbf {X}}_j|=n(n-1)^{j-1}\begin{vmatrix} s_2^2&s_2s_3\rho _{23}&\cdots&s_2s_j\rho _{2j}\\ s_2s_3\rho _{23}&s_3^2&\cdots&s_3s_j\rho _{3j}\\ \vdots&\vdots&\vdots&\vdots \\ s_2s_j\rho _{2j}&s_3s_j\rho _{3j}&\cdots&s_j^2\\ \end{vmatrix}, \end{aligned}$$

note that row l and column l are multiplied by $s_l$, using properties of the determinants

$$\begin{aligned} |{\mathbf {X}}_j^t{\mathbf {X}}_j|=n(n-1)^{j-1}s_2^2s_3^2\cdots s_j^2\begin{vmatrix} 1&\rho _{23}&\cdots&\rho _{2j}\\ \rho _{23}&1&\cdots&\rho _{3j}\\ \vdots&\vdots&\vdots&\vdots \\ \rho _{2j}&\rho _{3j}&\cdots&1\\ \end{vmatrix}=n(n-1)^{j-1}\prod _{l=2}^{j}s_{l}^2|R_j| \end{aligned}$$

on the other hand,

$$\begin{aligned} R_j=\begin{bmatrix} 1&{} \rho _{23}&{}\cdots &{}\rho _{2j}\\ \rho _{23}&{} 1&{}\cdots &{}\rho _{3j}\\ \vdots &{}\vdots &{}\vdots &{}\vdots \\ \rho _{2j}&{}\rho _{3j}&{}\cdots &{}1\\ \end{bmatrix}=\begin{bmatrix} R_i&{} R_{ij}\\ R_{ij}&{}R_{j-i}\\ \end{bmatrix} \end{aligned}$$

where

$$\begin{aligned} R_{ij}=\begin{bmatrix} \rho _{2j+1}&{} \rho _{3j+1}&{}\cdots &{}\rho _{ii+1}\\ \rho _{2j+2}&{} \rho _{3j+2}&{}\cdots &{}\rho _{ii+2}\\ \vdots &{}\vdots &{}\vdots &{}\vdots \\ \rho _{2j}&{}\rho _{3j+2}&{}\cdots &{}\rho _{ij}\\ \end{bmatrix}~~\text {and}~~R_{j-i}=\begin{bmatrix} 1&{} \rho _{i+2i+1}&{}\cdots &{}\rho _{ji+1}\\ \rho _{i+1i+2}&{} 1&{}\cdots &{}\rho _{ji+2}\\ \vdots &{}\vdots &{}\vdots &{}\vdots \\ \rho _{i+1j}&{}\rho _{i+2j}&{}\cdots &{}1\\ \end{bmatrix}. \end{aligned}$$

Now since ${\mathbf {X}}_j$ is a full rank matrix, it can be seen that

$$\begin{aligned} |R_j|=|R_i||R_{j-i}-R_{ij}^tR_i^{-1}R_{ij}| \end{aligned}$$

thus

$$\begin{aligned} b=\frac{|\mathbf {X}_j^t\mathbf {X}_j|}{|\mathbf {X}_i^t\mathbf {X}_i|}=(n-1)^{j-i}\left( \prod _{l=i+1}^{j}s^2_{l}\right) |R_{j-i}-R_{ij}^tR_i^{-1}R_{ij}| \end{aligned}$$