Assessing the Size of Model Misfit in Structural Equation Models

Abstract

When a statistically significant mean difference is found, the magnitude of the difference is judged qualitatively using an effect size such as Cohen’s d. In contrast, in a structural equation model (SEM), the result of the statistical test of model fit is often disregarded if significant, and inferences are drawn using “close” models retained based on point estimates of sample statistics (goodness-of-fit indices). However, when a SEM cannot be retained using a test of exact fit, all substantive inferences drawn from it are suspect. It is therefore important to determine the size of the model misfit. Standardized residual covariances and residual correlations provide standardized effect sizes of the misfit of SEM models. They can be summarized using the Standardized Root Mean Squared Residual (SRMSR) and the Correlation Root Mean Squared Residual (CRMSR) which can be used as overall effect sizes of the misfit. Statistical theory is provided that allows the construction of confidence intervals and tests of close fit based on the SRMSR and CRMSR. It is hoped that the use of standardized effect sizes of misfit will help reconcile current practices in SEM and elsewhere in statistics.

Appendix

1.1 Asymptotic Distribution of Standardized Residual Covariances and Residual Correlations

From standard theory (e.g., Satorra, 1989)

$$\begin{aligned} \sqrt{N}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}}}})\mathop =\limits ^a \mathbf{H} \sqrt{N}\left( {\mathbf{s}-{\varvec{{\upsigma }}}_0} \right) \end{aligned}$$

(34)

and the asymptotic distribution of the unstandardized (raw) residual covariances follows from (14) under parameter drift assumptions. The asymptotic distribution of the standardized residuals is obtained from a first-order Taylor expansion of ${{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})$ at $\mathbf{s}={\varvec{{\upsigma }}}$ and ${\hat{{\varvec{{\upsigma }}}}}={\varvec{{\upsigma }}}$

$$\begin{aligned}&{{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}}}})\approx \left. {{{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})} \right| _{\mathbf{s}={\varvec{{\upsigma }}} ,{\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}} } +\left. {\frac{\partial {{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})}{\partial \mathbf{{s}'}}} \right| _{\mathbf{s}={\varvec{{\upsigma }}} } (\mathbf{s}-{\varvec{{\upsigma }}} )+\left. {\frac{\partial {{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})}{\partial {{\hat{{\varvec{{\upsigma }}}}}}'}} \right| _{{\hat{{\varvec{{\upsigma }}} }} ={\varvec{{\upsigma }}}} ({\hat{{\varvec{{\upsigma }}} }}-{\varvec{{\upsigma }}} )\nonumber \\&\quad =\mathbf{G}(\mathbf{s}-{\varvec{{\upsigma }}} )-\mathbf{G}({\hat{{\varvec{{\upsigma }}} }}-{\varvec{{\upsigma }}} )=\mathbf{G}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }}), \end{aligned}$$

(35)

since $\left. {{{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})} \right| _{\mathbf{s}={\varvec{{\upsigma }}} ,{\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}} } =\mathbf{0}$, $\left. {\frac{\partial {{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})}{\partial \mathbf{{s}'}}} \right| _{\mathbf{s}={\varvec{{\upsigma }}} } =\mathbf{G}$ and $\left. {\frac{\partial {{\hat{\mathbf{G}}}}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }})}{\partial {{\hat{{\varvec{{\upsigma }}}}}}'}} \right| _{{\hat{{\varvec{{\upsigma }}}}}={\varvec{{\upsigma }}}} =-\mathbf{G}$. Equation (16) follows from (14), (34), and (35).

The asymptotic distribution of the residual correlations is obtained in a similar fashion. Using a first-order Taylor expansion of $\mathbf{r}-{\hat{{\varvec{\uprho }} }}$ at $\mathbf{s}={\varvec{{\upsigma }}}$ and ${\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}}$

$$\begin{aligned}&(\mathbf{r}-{\hat{{\varvec{\uprho }} }})\approx \left. {(\mathbf{r}-{\hat{{\varvec{\uprho }} }})} \right| _{\mathbf{s}={\varvec{{\upsigma }}} ,{\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}} } +\left. {\frac{\partial (\mathbf{r}-{\hat{{\varvec{\uprho }} }})}{\partial \mathbf{{s}'}}} \right| _{\mathbf{s}={\varvec{{\upsigma }}}} (\mathbf{s}-{\varvec{{\upsigma }}} )+\left. {\frac{\partial (\mathbf{r}-{\hat{{\varvec{\uprho }} }})}{\partial {{\hat{{\varvec{{\upsigma }}} }}}'}} \right| _{{\hat{{\varvec{{\upsigma }}}}}={\varvec{{\upsigma }}}} ({\hat{{\varvec{{\upsigma }}} }}-{\varvec{{\upsigma }}})\nonumber \\&\quad =\mathbf{F}(\mathbf{s}-{\varvec{{\upsigma }}} )-\mathbf{F}({\hat{{\varvec{{\upsigma }}} }}-{\varvec{{\upsigma }}} )=\mathbf{F}(\mathbf{s}-{\hat{{\varvec{{\upsigma }}} }}), \end{aligned}$$

(36)

where $\left. {(\mathbf{r}-{\hat{{\varvec{\uprho }}}})} \right| _{\mathbf{s}={\varvec{{\upsigma }}} ,{\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}} } =\mathbf{0}$, $\left. {\frac{\partial (\mathbf{r}-{\hat{{\varvec{\uprho }}}})}{\partial {\mathbf{s}'}}} \right| _{\mathbf{s}={\varvec{{\upsigma }}} } =\mathbf{F}$, $\left. {\frac{\partial (\mathbf{r}-{\hat{{\varvec{\uprho }} }})}{\partial {{\hat{{\varvec{{\upsigma }}} }}}'}} \right| _{{\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}}} =-\mathbf{F}$. The nonzero elements of F, ${\partial (r_{ij} -{\hat{\rho }}_{ij})}\big /{\partial {\hat{{\sigma } }}_{lk} }$ evaluated at ${\hat{{\varvec{{\upsigma }}} }}={\varvec{{\upsigma }}}$, are $-{{\sigma }_{ij}}\big /{\left( {2{\sigma }_{ii}^{3/2} {\sigma }_{jj}^{1/2} } \right) }=\frac{\rho _{ij} }{2{\sigma }_{jj} }$ if $i \ne { k, j = l}, -{{\sigma }_{ij} }\big /{\left( {2{\sigma }_{ii}^{1/2} {\sigma }_{jj}^{3/2} } \right) }=\frac{\rho _{ij} }{2{\sigma }_{ii} }$ if ${ i = k, j \ne l}$, and $1\big /{\left( {{\sigma }_{ii}^{1/2} {\sigma }_{jj}^{1/2}} \right) }$ if $(i, j) = (k, l)$ . Notice that F can be written as $\mathbf{F }=\mathbf{J G}$, where the nonzero elements of J are $-\frac{\rho _{ij} }{2}$ if ${i \ne k, j = l}$ or ${i = k, j \ne l}$, and 1 if $(i, j) = (k, l)$.

1.2 Asymptotic Mean and Variance of the Overall Effect Size Estimators

The asymptotic mean and variance of $T=T_{s}$ or $T_{r}$ under the parameter drift assumptions given in Eqs. (20) and (21) follow from standard results in quadratic forms of normal variates (e.g., Schott, 1997: Theorem 9.22).

The asymptotic mean and variance of $\tilde{{{\iota }} }$ follow from standard results on Taylor expansions of moments of functions of random variables: Let T be a random variable with ${\mu }_T$ and ${\sigma }_T^2$, then

$$\begin{aligned} E[f(T)]\approx & {} f({\mu }_T )+\frac{{f}''({\mu }_T )}{2}{{\sigma }}_T^2 , \end{aligned}$$

(37)

$$\begin{aligned} \hbox {var}[f(T)]\approx & {} \left[ {{f}'({\mu }_T )} \right] ^{2}{\sigma }_T^2 . \end{aligned}$$

(38)

For the $\tilde{{{\iota }}}_{s}$ based on the SRMR, $f(T)=\sqrt{\frac{T-c}{t}}$ and ${f}'(T)=\frac{1}{2t}\left( {\frac{T-c}{t}} \right) ^{-\frac{1}{2}}$ and ${f}''(T)=-\frac{1}{4t^{2}}\left( {\frac{T-c}{t}} \right) ^{-\frac{3}{2}}$. For the $\tilde{{{\iota }} }_\mathrm{r}$ based on the CRMR, $t-p$ is used instead of t. Equations (22) and (23) are obtained from (37) and (38) evaluating these derivatives at ${\mu }_T$ given in Eq. (20).

The asymptotic mean and variance of ${\hat{{{\iota }}}}=k^{-1}\tilde{{{\iota }} }$ are obtained from the asymptotic mean and variance of $\tilde{{{\iota }} }$ again using (37) and (38): $\mathrm{E}[{\hat{{{\iota }} }}]={{\iota }}, \hbox {var}[{\hat{{{\iota }} }}]=k^{-2}\hbox {var}[\tilde{{{\iota }}}]$.

1.3 Asymptotic Mean and Variance of the Sample SRMR and CRMR Under the Null Hypothesis of Exact Fit

Consider the statistic $T^{*}=\widehat{\hbox {SRMR}}\hbox { or }\widehat{\hbox {CRMR}}$ given in Eqs. (7) and (8). We write $T^{*}$ as $f(T)=\sqrt{T/t}$, where $T=T_{s}$ or $T_{r}$, and for $T_{r}\,t-p$ is used instead of t. Using (37) and (38) with ${f}'(T)=\frac{1}{2t}\left( {\frac{T}{t}} \right) ^{-\frac{1}{2}}$ and ${f}''(T)=-\frac{1}{4T^{2}}\left( {\frac{T}{t}} \right) ^{\frac{1}{2}}$,

$$\begin{aligned} \hbox {E}(T^{*})=\sqrt{\frac{{\mu }_T }{t}}\frac{8{\mu }_T^2 -{\sigma }_T^2 }{8{\mu }_T^2 }, \quad \hbox {var}(T^{*})=\frac{{\sigma }_T^2 }{4t\times {\mu }_T^2}. \end{aligned}$$

(39)

Now, under the null hypothesis of exact fit

$$\begin{aligned} \hbox {E}(T)={\mu }_T =\hbox {tr}({\varvec{\Xi }} ), \quad \hbox {var}(T)={\sigma }_{T}^{2}=2\hbox {tr}({\varvec{\Xi }} ^{2}). \end{aligned}$$

(40)

Equations (28) and (29) follow from (39) and (40).