Generalized SAMPLE SIZE Determination Formulas for Investigating Contextual Effects by a Three-Level Random Intercept Model

Abstract

Behavioral and psychological researchers have shown strong interests in investigating contextual effects (i.e., the influences of combinations of individual- and group-level predictors on individual-level outcomes). The present research provides generalized formulas for determining the sample size needed in investigating contextual effects according to the desired level of statistical power as well as width of confidence interval. These formulas are derived within a three-level random intercept model that includes one predictor/contextual variable at each level to simultaneously cover various kinds of contextual effects that researchers can show interest. The relative influences of indices included in the formulas on the standard errors of contextual effects estimates are investigated with the aim of further simplifying sample size determination procedures. In addition, simulation studies are performed to investigate finite sample behavior of calculated statistical power, showing that estimated sample sizes based on derived formulas can be both positively and negatively biased due to complex effects of unreliability of contextual variables, multicollinearity, and violation of assumption regarding the known variances. Thus, it is advisable to compare estimated sample sizes under various specifications of indices and to evaluate its potential bias, as illustrated in the example.

Appendix: Derivation of Standard Errors of Contextual Effects Estimates

Without loss of generality, assume an intercept $\gamma _0$ is 0. In matrix notation, the Eq. (1) can now be expressed as

$$\begin{aligned} \varvec{Y}=\varvec{\tilde{X}}\varvec{\beta }+{\tilde{\varvec{\epsilon }}}. \end{aligned}$$

(40)

Here, ${\varvec{{\beta }}}=(\gamma _1,\gamma _2,\gamma _3)'$, and ${{\varvec{{Y}}}}$ is an $(I\times J\times K)\times 1$ outcome vector. Its elements are arranged as $\varvec{Y}=(\varvec{Y'_{1}},\dots ,\varvec{Y'_{k}},\dots ,\varvec{Y'_{K}})'$, where $\varvec{Y_{k}}=(\varvec{Y'_{1k}},\dots ,\varvec{Y'_{jk}},\dots ,\varvec{Y'_{Jk}})'$ and $\varvec{Y_{jk}}=(Y_{1jk},\dots ,Y_{ijk},\dots ,Y_{Ijk})'$. As well, $\varvec{\tilde{X}}=(\varvec{X},\varvec{{\bar{X}}_{.jk}}, \varvec{{\bar{X}}_{..k}})$ is a $(I\times J\times K)\times 3$ predictor matrix; $\varvec{X}$ is a $(I\times J\times K)\times 1$ vector including predictors $X_{ijk}$, and its elements are arranged in similarly to those of $\varvec{Y}$. Then, $\varvec{{\bar{X}}_{.jk}}$ is an $(I\times J\times K)\times 1$ level-2 units mean vector, expressed as $\varvec{{\bar{X}}_{.jk}} =({\bar{X}}_{.11},\dots ,{\bar{X}}_{.1K},\dots ,{\bar{X}}_{.J1},\dots ,{\bar{X}}_{.JK})' \otimes \varvec{1_{I}}$, where $\otimes$ indicates the Kronecker product. Also, $\varvec{{\bar{X}}_{..k}}$ is an $(I\times J\times K)\times 1$ level-3 units mean vector, expressed as $\varvec{{\bar{X}}_{..k}} =({\bar{X}}_{..1},\dots ,{\bar{X}}_{..k},\dots ,{\bar{X}}_{..K})' \otimes \varvec{1_{IJ}}$. In addition, $\varvec{\tilde{\epsilon }}$ is the corresponding $(I\times J\times K)\times 1$ residual vector consisting of ${\tilde{e}}_{ijk}=e_{k}+e_{jk}+e_{ijk}$. From Eq. (7), it can be shown that $\varvec{\tilde{\epsilon }}$ is distributed as $\varvec{\tilde{\epsilon }}\sim N(\varvec{0},{\tilde{\varvec{\Sigma }}})$, where

$$\begin{aligned} {\varvec{\tilde{\Sigma }}}= & {} \varvec{I_K}\otimes \varvec{\Sigma }, \\ \varvec{\Sigma }= & {} \sigma _3^2\varvec{1_{IJ}}\varvec{1'_{IJ}}+\varvec{I_J}\otimes (\sigma _2^2\varvec{1_{I}}\varvec{1'_{I}})+\sigma _1^2\varvec{I_{IJ}}\nonumber \end{aligned}$$

(41)

$$\begin{aligned}= & {} \rho _2\varvec{1_{IJ}}\varvec{1'_{IJ}}+\varvec{I_J}\otimes [(\rho _1-\rho _2)\varvec{1_{I}}\varvec{1'_{I}}]+(1-\rho _1)\varvec{I_{IJ}}. \end{aligned}$$

(42)

Here, we assume that $\sigma _1^2\ge 0$, $\sigma _2^2\ge 0$, and $\sigma _3^2\ge 0$, and that the inverse matrix of $\varvec{\Sigma }$ (denoted as $\varvec{\Sigma ^{-1}}$) exists. Let the diagonal elements of $\varvec{\Sigma ^{-1}}$ be $\sigma ^{(1)}$, off-diagonal elements denoting the same level-2 and level-3 unit in $\varvec{\Sigma ^{-1}}$ be $\sigma ^{(2)}$, and off-block elements denoting the same level-3 unit in $\varvec{\Sigma ^{-1}}$ be $\sigma ^{(3)}$. Comparing the left and right sides of the identity $\varvec{\Sigma }\varvec{\Sigma ^{-1}}=\varvec{I}$, the following relations are obtained:

$$\begin{aligned} \sigma ^{(1)}= & {} \frac{f(\rho _1-\rho _2)(I-1)+(1-\rho _1)(f-\rho _2)}{f(1-\rho _1)[I(\rho _1-\rho _2)+(1-\rho _1)]} \nonumber \\ \sigma ^{(2)}= & {} -\frac{f(\rho _1-\rho _2)+\rho _2(1-\rho _1)}{f(1-\rho _1)[I(\rho _1-\rho _2)+(1-\rho _1)]}\nonumber \\ \sigma ^{(3)}= & {} -\frac{\rho _2}{f[I(\rho _1-\rho _2)+(1-\rho _1)]} \end{aligned}$$

(43)

Using the generalized least squares estimators, a sample distribution of $\hat{\varvec{\beta }}$ can be expressed as

$$\begin{aligned} \hat{\varvec{\beta }}\sim N( ({{\tilde{\varvec{X}}}^{\prime }}{\varvec{{\tilde{\Sigma }^{-1}}}}\tilde{\varvec{X}})^{-1}{\tilde{\varvec{X}}}^{\prime }{\varvec{{\tilde{\Sigma }^{-1}}}}\varvec{Y}, ({{\tilde{\varvec{X}}}^{\prime }}{\varvec{{\tilde{\Sigma }^{-1}}}}\tilde{\varvec{X}})^{-1}). \end{aligned}$$

(44)

Then $se({\hat{\gamma }}_2)$ and $se({\hat{\gamma }}_3)$ can be evaluated by the square root of (2, 2) and (3, 3) elements of $({\tilde{\varvec{X}}}^{\prime }{\varvec{{\tilde{\Sigma }^{-1}}}}\tilde{\varvec{X}})^{-1}=[{\tilde{\varvec{X}}}^{\prime }({\varvec{I_K}\otimes \varvec{\Sigma ^{-1}}})\tilde{\varvec{X}}]^{-1} =\varvec{\Sigma ^*}^{-1}$, respectively. Here, $\varvec{\Sigma ^*}={\tilde{\varvec{X}}}^{\prime }({\varvec{I_K}\otimes \varvec{\Sigma ^{-1}}})\tilde{\varvec{X}}$.

$\sigma ^*_{11}$, a (1,1) element of $\varvec{\Sigma ^*}$, can be calculated as

$$\begin{aligned} \sigma ^*_{11}= & {} \varvec{X'}(\varvec{I_K}\otimes \varvec{\Sigma ^{-1}})\varvec{X} \nonumber \\= & {} \sigma ^{(1)}\sum _{i}\sum _{j}\sum _{k}X^2_{ijk}+\,\sigma ^{(2)}\sum _{i}\sum _{i'\ne i}\sum _{j}\sum _{k}X_{ijk}X_{i'jk}+\sigma ^{(3)}\sum _{i}\sum _{i'}\sum _{j}\sum _{j'\ne j}\sum _{k}X_{ijk}X_{i'j'k}\nonumber \\= & {} \sigma ^{(1)}\sum _{i}\sum _{j}\sum _{k}X^2_{ijk}\nonumber \\&+\, \sigma ^{(2)}(I-1)\left[ IJ\sum _{k}({\bar{X}}_{..k}-{\bar{X}}_{...})^2+I\sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{..k})^2-\frac{\sum _{i}\sum _{j}\sum _{k}(X_{ijk}-{\bar{X}}_{.jk})^2}{I-1}\right] \nonumber \\&+\, \sigma ^{(3)}I(J-1)\left[ IJ\sum _{k}({\bar{X}}_{..k}-{\bar{X}}_{...})^2-\frac{I\sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{..k})^2}{J-1}\right] . \end{aligned}$$

(45)

Here, ${\bar{X}}_{...}$ is the overall mean of the predictor. $\sigma ^*_{22}$, a (2,2) element of $\varvec{\Sigma ^*}$, can be calculated as

$$\begin{aligned} \sigma ^*_{22}= & {} [({\bar{X}}_{.11},\dots ,{\bar{X}}_{.1K},\dots ,{\bar{X}}_{.J1},\dots ,{\bar{X}}_{.JK})\otimes \varvec{{1'}{}_{I}}]\nonumber \\&(\varvec{I_K}\otimes \varvec{\Sigma ^{-1}})[({\bar{X}}_{.11},\dots ,{\bar{X}}_{.1K},\dots ,{\bar{X}}_{.J1},\dots ,{\bar{X}}_{.JK})'\otimes \varvec{{1}_{I}}]\nonumber \\= & {} \sigma ^{(1)}I\sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{...})^2+\sigma ^{(2)}I(I-1)\sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{...})^2\nonumber \\&+\sigma ^{(3)}I^2(J-1)\sum _{j}\sum _{j'\ne j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{...})({\bar{X}}_{.j'k}\,-\,{\bar{X}}_{...})\nonumber \\= & {} \left[ \sigma ^{(1)}I+\sigma ^{(2)}I(I-1)\right] \left[ \sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{..k})^2+J\sum _{k}({\bar{X}}_{..k}-{\bar{X}}_{...})^2\right] \nonumber \\&+\, \sigma ^{(3)}I(J-1)\left[ IJ\sum _{k}({\bar{X}}_{..k}-{\bar{X}}_{...})^2-\frac{I\sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{..k})^2}{J-1}\right] . \end{aligned}$$

(46)

$\sigma ^*_{33}$, a (3,3) element of $\varvec{\Sigma ^*}$, can be calculated as

$$\begin{aligned} \sigma ^*_{33}= & {} [({\bar{X}}_{..1},\dots ,{\bar{X}}_{..k},\dots ,{\bar{X}}_{..K})\otimes \varvec{{1'}{}_{IJ}}]\nonumber \\&(\varvec{I_K}\otimes \varvec{\Sigma ^{-1}})[({\bar{X}}_{..1},\dots ,{\bar{X}}_{..k},\dots ,{\bar{X}}_{..K})'\otimes \varvec{1_{IJ}}]\nonumber \\= & {} [\sigma ^{(1)}+(I-1)\sigma ^{(2)}+I(J-1)\sigma ^{(3)}][({\bar{X}}_{..1},\dots ,{\bar{X}}_{..k},\dots ,{\bar{X}}_{..K})\otimes \varvec{{1'}{}_{IJ}}]\nonumber \\&[({\bar{X}}_{..1},\dots ,{\bar{X}}_{..k},\dots ,{\bar{X}}_{..K})'\otimes \varvec{1_{IJ}}]\nonumber \\= & {} IJf^{-1}({\bar{X}}^2{}_{..1}+\dots +{\bar{X}}^2{}_{..k}+\dots +{\bar{X}}^2{}_{..K})\nonumber \\= & {} \frac{IJ\sum _{k}{\bar{X}}^2{}_{..k}}{f}. \end{aligned}$$

(47)

Considering the identical equations of $I\sum _{j}\sum _{k}{{\bar{X}}^2{}_{.jk}}=\sum _{i}\sum _{j}\sum _{k}X_{ijk}{{\bar{X}}_{.jk}}$, it follows that $\sigma ^*_{12}$, a (1,2) element of $\varvec{\Sigma ^*}$, is equal to $\sigma ^*_{22}$. Likewise, $\sigma ^*_{13}$ and $\sigma ^*_{23}$, which are, respectively, (1,3) and (2,3) elements of $\varvec{\Sigma ^*}$, are equal to $\sigma ^*_{33}$. Thus, elements in $\varvec{\Sigma ^*}$ show the following inclusion relation among levels:

$$\begin{aligned} \varvec{\Sigma ^*}=\left( \begin{array}{ccc} \sigma ^*_{11} &{} \sigma ^*_{22} &{} \sigma ^*_{33} \\ \sigma ^*_{22} &{} \sigma ^*_{22} &{} \sigma ^*_{33} \\ \sigma ^*_{33} &{} \sigma ^*_{33} &{} \sigma ^*_{33} \end{array} \right) . \end{aligned}$$

(48)

Assume here ${\bar{X}}_{...}=0$ without loss of generality of discussion. Since $\sum _{i}\sum _{j}\sum _{k}(X_{ijk}-{\bar{X}}_{...})^2=\sum _{i}\sum _{j}\sum _{k}X^2{}_{ijk}=IJK\sigma _x^2$, $\eta _3^2=\frac{\sum _{k}{\bar{X}}^2{}_{..k}}{K\sigma _x^2}$ from Eq. (12). Then, $\sum _{k}{\bar{X}}^2{}_{..k}$ can be expressed using $\eta _3^2$ as

$$\begin{aligned} \sum _{k}{\bar{X}}^2{}_{..k}=K\sigma _x^2\eta _3^2. \end{aligned}$$

(49)

From this result and the relations (11)–(14), the following relations are also obtained:

$$\begin{aligned} \sum _{i}\sum _{j}\sum _{k}(X_{ijk}-{\bar{X}}_{..k})^2= & {} IJK\sigma _x^2(1-\eta _3^2), \end{aligned}$$

(50)

$$\begin{aligned} \sum _{j}\sum _{k}({\bar{X}}_{.jk}-{\bar{X}}_{..k})^2= & {} JK\sigma _x^2\eta _2^2(1-\eta _3^2), \end{aligned}$$

(51)

$$\begin{aligned} \sum _{i}\sum _{j}\sum _{k}(X_{ijk}-{\bar{X}}_{.jk})^2= & {} IJK\sigma _x^2(1-\eta _2^2)(1-\eta _3^2). \end{aligned}$$

(52)

Then, it is possible to further simplify $\sigma ^{*}_{11}$, $\sigma ^{*}_{22}$, and $\sigma ^{*}_{33}$ as

$$\begin{aligned} \sigma ^{*}_{11}= & {} \sigma ^{(1)}IJK+\sigma ^{(2)}(I-1)IJK\sigma _x^2\left[ \eta _3^2+\eta _2^2(1-\eta _3^2)-\frac{(1-\eta _3^2)(1-\eta _2^2)}{I-1}\right] +\, \sigma ^{(3)}I^2JK(J-1)\sigma _x^2\left[ \eta _3^2-\frac{\eta _2^2(1-\eta _3^2)}{J-1}\right] \nonumber \\= & {} { IJK}\sigma _x^2\left[ \sigma ^{(1)}+\sigma ^{(2)}(I\eta _3^2+I\eta _2^2-I\eta _2^2\eta _3^2-1)+\sigma ^{(3)}I\left[ (J-1)\eta _3^2-\eta _2^2(1-\eta _3^2)\right] \right] \nonumber \\= & {} { IJK}\sigma _x^2\frac{(1-\eta _3^2)\rho _2IJ[(1-\eta _2^2)(\rho _1-\rho _2)I+(1-\rho _1)]+[I(\rho _1-\rho _2)+(1-\rho _1)] [(1-\eta _2^2)(1-\eta _3^2)(\rho _1-\rho _2)I+(1-\rho _1)]}{(1-\rho _1)[I(\rho _1-\rho _2)+(1-\rho _1)]f},\nonumber \\\end{aligned}$$

(53)

$$\begin{aligned} \sigma ^{*}_{22}= & {} [\sigma ^{(1)}I+\sigma ^{(2)}I(I-1)][JK\sigma _x^2\eta _2^2(1-\eta _3^2)+JK\sigma _x^2\eta _3^2]+\, \sigma ^{(3)}I(J-1)\left[ IJK\sigma _x^2\eta _3^2-IJK\sigma _x^2\frac{\eta _2^2(1-\eta _3^2)}{J-1}\right] \nonumber \\= & {} { IJK}\sigma _x^2\left[ [\sigma ^{(1)}+\sigma ^{(2)}(I-1)][\eta _2^2(1-\eta _3^2)+\eta _3^2]+\sigma ^{(3)}I[(J-1)\eta _3^2-\eta _2^2(1-\eta _3^2)]\right] \nonumber \\= & {} { IJK}\sigma _x^2\frac{\eta _2^2(1-\eta _3^2)\rho _2IJ+(\eta _2^2+\eta _3^2-\eta _2^2\eta _3^2)[I(\rho _1-\rho _2)+(1-\rho _1)]}{[I(\rho _1-\rho _2)+(1-\rho _1)]f} \end{aligned}$$

(54)

$$\begin{aligned} \sigma ^{*}_{33}= & {} \frac{IJ\sum _{k}{\bar{X}}^2{}_{..k}}{f}=\frac{IJK\eta _3^2\sigma _x^2}{f}, \end{aligned}$$

(55)

where $f=1+I(J-1)\rho _2+(I-1)\rho _1=1/[\sigma ^{(1)}+(I-1)\sigma ^{(2)}+I(J-1)\sigma ^{(3)}]$. From these results and derived structures of $\varvec{\Sigma ^*}$, the standard errors $se({\hat{\gamma }}_1)$, $se({\hat{\gamma }}_2)$, and $se({\hat{\gamma }}_3)$ can be calculated as

$$\begin{aligned} se({\hat{\gamma }}_1)= & {} \sqrt{\frac{1}{det(\varvec{\Sigma ^{*}})}(\sigma ^{*}_{22}\sigma ^{*}_{33}-\sigma ^{*}_{33}\sigma ^{*}_{33})}= \sqrt{\frac{1-\rho _1}{IJK(1-\eta _2^2)(1-\eta _3^2)\sigma _x^2}},\ \end{aligned}$$

(56)

$$\begin{aligned} se({\hat{\gamma }}_2)= & {} \sqrt{\frac{1}{det(\varvec{\Sigma ^{*}})}(\sigma ^{*}_{11}\sigma ^{*}_{33}-\sigma ^{*}_{33}\sigma ^{*}_{33})}= \sqrt{\frac{I[(1-\eta _2^2)(\rho _1-\rho _2)]+(1-\rho _1)}{IJK(1-\eta _2^2)\eta _2^2(1-\eta _3^2)\sigma _x^2}}, \end{aligned}$$

(57)

$$\begin{aligned} se({\hat{\gamma }}_3)= & {} \sqrt{\frac{1}{det(\varvec{\Sigma ^{*}})}(\sigma ^{*}_{11}\sigma ^{*}_{22}-\sigma ^{*}_{22}\sigma ^{*}_{22})}\nonumber \\= & {} \sqrt{\frac{IJ\rho _2\eta _2^2(1-\eta _3^2)+[I(\rho _1-\rho _2)+1-\rho _1](\eta _2^2+\eta _3^2-\eta _2^2\eta _3^2)}{IJK\eta _2^2(1-\eta _3^2)\eta _3^2\sigma _x^2}}. \end{aligned}$$

(58)

Here, det($\cdot$) denotes the determinant, and det($\varvec{\Sigma ^{*}})=\sigma ^{*}_{33}(\sigma ^{*}_{22}-\sigma ^{*}_{33})(\sigma ^{*}_{11}-\sigma ^{*}_{22})$.