Investigating the performance of level-specific fit indices in multilevel confirmatory factor analysis with dichotomous indicators: A Monte Carlo study

Abstract

We conducted a Monte Carlo study to examine the performance of level-specific χ² test statistics and fit regarding their capacity to determine model fit at specific levels in multilevel confirmatory factor analysis with dichotomous indicators. Five design factors—numbers of groups (NG), group size (GS), intra-class correlation (ICC), thresholds of dichotomous indicators (THR), and factor loadings (FL)—were considered in this study. According to our simulation results, we recommend that practitioners should be aware that the performance of between-level-specific (b-l-s) χ² and fit indices was mainly influenced by ICC and FL, followed by NG. At the same time, THR could slightly weigh in the performance of b-l-s fit indices in some conditions. Both b-l-s χ² and fit indices were more promising indicators to correctly indicate model fit when ICC or FL increased. A small to medium NG (50–100) might be sufficient for b-l-s χ² and fit indices only if both ICC and factor loadings were high, while in remaining conditions, an NG of 200 was needed. Moreover, practitioners could use within-level-specific (w-l-s) χ² and fit indices (except for RMSEA_W) along with traditional cut-off values to evaluate within-level models comprising dichotomous indicators. W-l-s χ² and fit indices were more promising to determine model fit when FL increased. THR had a slight impact and could weigh in the performance of ${\chi}_W^2$, RMSEA_W, CFI_W, and TLI_W. Unfortunately, RMSEA_W was heavily affected by FL and THR and could determine model fit only when FL was high and THR was symmetric.

Introduction

Multilevel confirmatory factor analysis (MCFA), or multilevel measurement modeling, is an imperative statistical approach to validate latent constructs that underlie multiple item responses collected in multilevel settings (e.g., students nested within schools). MCFA has been widely used in psychological and educational research, and relevant reporting guidelines have been made available for substantive researchers (Kim et al., 2016). Unlike conventional confirmatory factor analysis, the construct of interest in MCFA could be at multiple levels, resulting in challenges for substantive researchers in model specification, evaluation, and interpretation. Due to this complexity, extensive efforts have been devoted to providing instructions for appropriate specification and interpretation of MCFA (Stapleton, McNeish et al., 2016a; Stapleton, Yang et al., 2016b).

Regarding model evaluation, a body of research has shown that traditional fit indices [e.g., root mean square error of approximation (RMSEA), comparative fit index (CFI), Tucker–Lewis Index (TLI), and standardized root mean square residual (SRMR)] are not sensitive to misspecified between-level models in multilevel structural equation modeling (MSEM) (Hsu et al., 2017; Padgett & Morgan, 2020). For this reason, researchers have advocated the use of level-specific (l-s) fit indices for evaluating the within-level model and the between-level model separately (Hox, 2010; Hsu et al., 2017; Rappaport et al., 2020; Ryu, 2014; Ryu & West, 2009; Schermelleh-Engel et al., 2014; Wu et al., 2017). According to Ryu and West (2009), the l-s fit indices could be straightforwardly computed by the partially saturated-model method (PS method). Note that the existing guidelines for using level-specific fit indices are based on population multilevel measurement models with continuous indicators, and few address ordered categorical variables (Padgett & Morgan, 2020). Yet it is unclear whether these guidelines could be applicable to models with dichotomous indicators.

To shed light on this issue, this study endeavors to examine the performance of level-specific χ² test statistics and fit indices derived from the PS method in terms of their sensitivity to lack of fit at specific levels in MCFA with dichotomous indicators. In addition, the effectiveness of alternative level-specific fit indices obtained from Mplus—SRMR for the within-level model (SRMR_w) and for the between-level model (SRMR_B)—was compared with PS-level-specific fit indices.

MCFA with dichotomous indicators in multilevel structural equation modeling

For simplicity, we consider a two-level single-factor model. Let Y_pig denote the p^th dichotomous indicator (i.e., latent variable indicator) for individual i nested within g group (p = 1…P dichotomous indicators, i = 1…N individuals, and g = 1…G groups),

$${Y}_{pig}=\left\{\begin{array}{c}1, if\ {y}_{pig}^{\ast }>\tau \\ {}0, if\ {y}_{pig}^{\ast}\le \tau \end{array}\right.$$

(1)

The equation expresses a threshold model which assumes that underlying the dichotomous indicator Y_pig is a normally distributed continuous latent variable${y}_{pig}^{\ast }$, which can determine the category of dichotomous indicator by the threshold (τ) (Asparouhov & Muthen, 2007; Bollen, 2002). That is, the indicators of interest are conceptualized as continuous, but the format of response to each indicator is in a restrictive, dichotomous scale (Bollen, 2002). For example, if the i^th individual falls short of the threshold, the response of this individual would be 0. If the i^th individual passes the threshold, the response of this individual would be 1.

Using similar notations to those used by Padgett and Morgan (2021), in this section, we outline a two-level measurement model with dichotomous indicators, as illustrated in Fig. 1. Note that the model in Fig. 1 was also adopted as a population model for simulated dataset generation in the current study. Using the between-and-within specification approach (Hox, 2010; Muthen, 1994), the covariance structure is partitioned into a within-level component (denoted by a W subscript) and a between-level component (denoted by a B subscript). Separate models are specified for each component. The within-level component captures individual-level variation, while the between-level component captures variation between groups.

Fig. 1

A two-level data generating model with two factors at each level

Let y_ig denote the p-dimensional response vector for student i in group g. The response vector y_ig is decomposed as:

$${y}_{ig}=\mu +{y}_{w_{ig}}+{y}_{B_g}$$

(2)

where μ represents the grand mean, and ${y}_{w_{ig}}$ and ${y}_{B_g}$ are independent within-level and between-level components, respectively. The measurement model at the within-level is given by the equation:

$${y}_{ig}={\mu}_g+{\varLambda}_W{\eta}_{W_{ig}}+{\epsilon}_{W_{ig}}$$

(3)

where Λ_W is the p × 2 factor loadings matrix for the within-level latent factor (${\eta}_{W_{ig}}$), ${\eta}_{W_{ig}}$ vector is the distributed multivariate normal with an expectation of zero and is a 2 × 2 covariance matrix Ψ_W. The ${\epsilon}_{W_{ig}}$ is multivariate normal distributed with an expectation of zero and p × p diagonal covariance matrix Θ_W, with error terms along the diagonal.

The measurement model at the between-level is given by

$${\mu}_g=\mu +{\varLambda}_B{\eta}_{B_g}+{\epsilon}_{B_g}$$

(4)

Here, Λ_B, ${\eta}_{B_g}$, and ${\epsilon}_{B_g}$ are the between-level terms corresponding to the within-level terms Λ_W, ${\eta}_{W_{ig}}$, and ${\epsilon}_{W_{ig}}$. Moreover, the covariance matrices Ψ_B and Θ_B are the between-level counterparts to the within-level covariance matrices Ψ_W and Θ_W. We can obtain Eq. (5) after combining Eqs. (3) and (4):

$${y}_{ig}=\mu +{\varLambda}_W{\eta}_{W_{ig}}+{\varLambda}_B{\eta}_{B_g}+{\epsilon}_{W_{ig}}+{\epsilon}_{B_g}$$

(5)

Estimator

The diagonally weighted least squares (DWLS) estimator is recommended for single-level confirmatory factor analysis (CFA) with categorical indicators (DiStefano & Morgan, 2014; Forero et al., 2009) and multilevel CFA due to its ability to identify the correct model specification (Padgett & Morgan, 2020). The DWLS estimator was based on polychoric correlation and the inverse of the asymptotic covariance matrix W^-1 of the sample variances and covariances as a weight matrix. Because the estimation of W^-1 is quite unstable when the sample size is small, the DWLS uses only the diagonal elements of W in model fitting and uses full W to obtain standard errors and χ² values (Jöreskog & Sörbom, 1996). As a result, the DWLS produces robust standard errors and χ² values (Flora & Curran, 2004; Muthen, 1993). Finney et al., (2006) suggested that when fewer than five categories are used, the DWLS estimator resulted in robust parameter estimates, standard errors, and fit indices for models with categorical nature of the data. In addition, the DWLS estimator has been found to perform well with small sample sizes and large models (Flora & Curran, 2004; Yang-Wallentin et al., 2010). Beauducel and Herzberg (2006) found that DWLS produced fit indices (root mean square error of approximation [RMSEA], comparative fit index [CFI], Tucker-Lewis Index [TLI]) that adequately indicated correctly specified models. However, when data were non-normally distributed in two to four categories, Bandalos (2008) found that robust DWLS-based RMSEA and CFI inadequately identified poorly misspecified models. Note that the DWLS estimator is known as weighted least squares mean and variance adjusted (WLSMV) estimator in Mplus. In the present study, the DWLS estimator was adopted for analyzing simulated datasets by using the command “estimator = WLSMV” in Mplus.

Cut-off values for using fit indices on CFA with dichotomous indicators

Several simulation studies have examined whether Hu and Bentler’s (1999) conventional cut-off values for traditional fit indices (i.e., RMSEA < .06; CFI and TLI > .95; SRMR < .08) can be applied similarly when DWLS is used to ordered categorical data. In general, prior studies suggested that conventional cut-off values should be applied with careful consideration of the data’s characteristics, such as sample size, asymmetry of categorical data, and number of categories (DiStefano & Morgan, 2014; Garrido, Abad, & Ponsoda, 2016; Nye & Drasgow, 2011; Xia & Yang, 2019). To the best of our knowledge, Padgett and Morgan (2021) is the only study that provided recommended cut-off criteria for using traditional fit indices in MCFA with categorical indicators. Specifically, Padgett and Morgan found that CFI, TLI, and RMSEA were primarily influenced by within-level misspecification, but partially influenced by between-level misspecification. In addition, the performance of the three fit indices was impacted by the sample size at the between-level (N₂) when DWLS was used. Although they provide recommended cut-off criteria for CFI and TLI (> .98 if N₂ < 100; > .97 if N₂ ≥ 100), and RMSEA (< .02 regardless of N₂), they also cautioned that those fit indices should be used only to provide weak evidence for some type of misspecification. Alternatively, SRMR_w and SRMR_B may provide within-level and between-level model-data fit, respectively. When DWLS is used, SRMR_w needs a cut-off value of < .05 if N₂ < 100, and a cut-off value of < .04 if N₂ ≥ 100. On the other hand, SRMR_B is suggested to have a cut-off value of < .06 if N₂ ≥ 100 and should not be used if N₂ < 100.

Model evaluation: PS-level-specific fit indices

Numerous simulation studies have indicated that level-specific tests of exact fit (i.e., χ² test statistics) and fit indices derived from the partially saturated-model method were recommended for detecting misspecification in MCFA (Hsu et al., 2017; Lee & Sohn, 2022; Rappaport et al., 2020; Ryu, 2011, 2014; Ryu & West, 2009; Schermelleh-Engel et al., 2014). Using the PS method, researchers can first derive the between-level-specific (b-l-s) χ² test statistics (${\chi}_{PS\_B}^2$). Specifically, ${\chi}_{PS\_B}^2$ can be derived by specifying a hypothesized between-level model and saturating the within-level model (i.e., correlating all observed variables). A saturated within-level models can be seen as a just-identified model with zero degrees of freedom, and thus has a χ² test statistic equal to zero. Consequently, ${\chi}_{PS\_B}^2$ only reflects the model fit of the hypothesized between-level model. Since fit indices are a function of χ² test statistics, researchers can then compute b-l-s fit indices (RMSEA_{PS_B}, CFI_{PS_B}, TLI_{PS_B}) using the value of ${\chi}_{PS\_B}^2$.

In the same manner, within-level-specific (w-l-s) χ² test statistics (${\chi}_{PS\_W}^2$) can be derived by specifying a hypothesized within-level model and saturating the between-level model. The w-l-s fit indices (RMSEA_{PS_W}, CFI_{PS_W}, TLI_{PS_W}) can be computed using the value of ${\chi}_{PS\_W}^2$. The formulas for computing l-s fit indices are identical to those in Ryu and West (2009) and Hsu, Lin, Skidmore, and Kim (2018) studies (see Appendix A). Additionally, the performance of the aforementioned l-s fit indices was compared with that of two alternative l-s fit indices, SRMR_W and SRMR_B, which are computed based on the discrepancy between the sample covariance and the corresponding model-implied covariance. Both SRMR_W and SRMR_B are available in the Mplus model solution output.

Previous simulation studies on L-s fit indices

Prior simulation studies have investigated the performance of l-s fit indices when the multivariate normality assumption was met in MCFA. Ryu and West (2009) examined the effectiveness of b-l-s fit indices (RMSEA_{PS_B}, CFI_{PS_B}) and w-l-s fit indices (RMSEA_{PS_W}, CFI_{PS_W}) using a population MCFA model with continuous indicators where multivariate normality was assumed. The intraclass correlation coefficient (ICC) level in the population model was fixed to 0.5. Ryu and West (2009) found that both types of fit indices can correctly indicate good and poor model fit in various sample size conditions (numbers of groups = 50, 100, 200, and 1000, and group size = 20, 50, 100). Ryu and West (2009), however, also found that a sample size of 50 groups could cause nonconvergence problems. Ryu and West’s findings were validated by Rappaport et al. (2020).

Hsu et al. (2017) extended the findings of Ryu and West (2009) by investigating the performance of l-s fit indices when ICC was smaller than 0.5. Hsu et al. (2017) found that the performance of w-l-s fit indices (RMSEA_{PS_W}, CFI_{PS_W}, TLI_{PS_W}, and SRMR_w) was barely influenced by ICC, while b-l-s fit indices (RMSEA_{PS_B}, CFI_{PS_B}, TLI_{PS_B}, and SRMR_B) were less-promising indicators to correctly indicate good or poor model fit. Similarly, Lee and Sohn (2022) found that w-l-s fit indices were sensitive to detecting misspecified within-level models and were less impacted by ICC and sample size. In addition, Lee and Sohn discovered that RMSEA_{PS_B} and SRMR_B were more promising for detecting misspecified between-level models with an increase in ICC, while CFI_{PS_B} and TLI_{PS_B} were also influenced by ICC, but the influence was moderated by the type of misspecifications in models (e.g., misspecification in factor cross-loadings or factor covariance). In summary, previous studies have considered various design factors, such as numbers of groups, group size, ICC, and type of misspecifications.

To the best of our knowledge, the performance of l-s fit indices in MCFA with categorical indicators has not been well examined in prior research. Hsu (2009) examined the sensitivity of SRMR_W and SRMR_B in MCFA with dichotomous indicators. Hsu (2009) found that SRMR_W can correctly indicate good model fit (i.e., type I error rate < .05) and poor model fit (i.e., statistical power > .80) due to an intentional misspecification in factor covariance. The type I error rate of SRMR_B, however, tended to be higher across different conditions, and the statistical power was less satisfying when ICC was low. Similar findings about SRMR_W and SRMR_B were revealed in Navruz’s (2016) study.

The present study

To date, no studies have attempted to evaluate the performance of l-s fit indices derived from the PS method in MCFA with categorical indicators. To address this concern, the present study attempted to evaluate the performance of commonly used l-s fit indices using a population model with dichotomous indicators. The findings of this study may shed some light on the practice of model evaluation when categorical indicators are used in MCFA.

The design factors considered in the present study included numbers of groups, group size, ICC, as well as two other factors, thresholds of dichotomous indicators and factor loadings, which have not been widely investigated in previous simulation studies focusing on effectiveness of l-s fit indices. Asymmetric thresholds of dichotomous indicators could occur in real-world situations, and prior studies have discovered the impact of asymmetric thresholds on the performance of several scaled or adjusted χ² statistics (hereafter called adjusted χ² statistics). For example, Rhemtulla, Brosseau-Liard, and Savalei (2012) examined three different conditions of thresholds (50:50, 60:40, and 80:20) and found that adjusted χ² statistics had relatively low statistical power in detecting serious model misspecification with asymmetric thresholds and small samples. In addition, type I error rates were reasonable (smaller than .05) when the threshold was symmetric, but extreme asymmetry thresholds (80:20) could cause high type I error rates. Savalei and Rhemtulla (2013) conducted simulations based on a two-factor model with dichotomous indicators using a DWLS estimator. They examined the difference between symmetric thresholds (50:50) and asymmetric thresholds (64:36 and 85:15) on adjusted χ² statistics examined in Rhemtulla et al. (2012). Results suggested that the performance of adjusted χ² statistics decreases as thresholds become more asymmetric. To the best of our knowledge, the impact of asymmetric thresholds on l-s fit indices has not been well documented. Because fit indices were a function of χ² statistics, the performance of fit indices could very likely be influenced by the thresholds of dichotomous indicators. For this reason, we examined the impact of asymmetric thresholds on level-specific fit indices in MCFA with dichotomous indicators.

In addition, previous simulation studies (e.g., Forero et al., 2009; Garrido et al., 2016; Heene et al., 2011; Nestler, 2013) have consistently shown that CFAs with low factor loadings resulted in less adequate model evaluation results. Nestler (2013) examined the impact of factor loadings—high (0.70), medium (0.55), and low (0.40)—on χ² statistics with a two-factor CFA with dichotomous indicators. Nestler found that when factor loadings were high, the statistical power of χ² test statistics was satisfying (~>.80), regardless of sample size. However, when factor loadings were medium or low, a sample size of 250 or 500, respectively, was needed to retain satisfactory statistical power. Furthermore, previous simulation studies have shown that the magnitudes of factor loadings had an impact on the performance of traditional fit indices. For example, Heene et al. (2011) examined how the performance of the χ² test statistic, RMSEA, CFI, and SRMR could be influenced by three levels of factor loadings [low ~(0.30, 0.50); medium ~(0.50, 0.70); high ~(0.70, 0.90)]. Heene et al., 2011 found that decreasing factor loadings led to decreasing values of the χ² test statistic and three fit indices, altering the statistical power to detect misspecified models. In general, low factor loadings resulted in decreasing values of χ² test statistic and RMSEA, CFI, and SRMR. As a result, misspecification was often not detected by the χ² test statistic, RMSEA, or SRMR when factor loadings were low. However, high factor loadings can cause the rejection of just slightly misspecified models (cf. Savalei, 2012). In contrast, Heene et al. , 2011 found that the CFI tended to exhibit poorer fit for models with low factor loadings. The reason is that models with lower factor loadings held lower covariances between the observed variables. As a result, the distance between the hypothesized model and the baseline null model would be reduced, resulting in lower CFI values (Garrido et al., 2016). The impact of low factor loadings on CFI can be extended to TLI because both fit indices are a function of the distance between the hypothesized model and the baseline null model. To date, few efforts have been made to study the impact of factor loadings on l-s fit indices in MCFA with dichotomous indicators. Our study aimed to bridge this knowledge gap.

Note that although we were aware that type of misspecification (MT) can be manipulated as a design factor, our study only considered one MT condition, which was the misspecification in factor covariance. Our selected misspecification scenario is important for applied researchers who wish to verify the construct validity of their instruments. We did not include the scenario of misspecification in cross-loadings as another MT condition because we think misspecification in cross-loadings is a highly complex topic (e.g., over-specification, under-specification, magnitudes of factor cross-loadings) which deserves a new study to comprehensively investigate this issue.

Method

A Monte Carlo study was conducted to evaluate the performance of l-s fit indices in MCFA with dichotomous indicators. The five design factors examined in this study were numbers of groups, group size, intraclass correlation coefficient, thresholds of dichotomous indicators, and factor loadings.

Population model

In the current study, the population model (see Fig. 1) for simulation data generation was based on the population model presented in Hsu et al.’s (2017) study. The population model was a two-level measurement model with two within-level factors (η_W1and η_W2) and two between-level factors (η_B1and η_B2). At the within-level, five dichotomous observed indicators were loaded on each factor. Parameters in the within-level model for simulation data generation were factor loadings = 0.70, factor variances = 1.00, and factor covariance = 0.30. The residual variance parameters cannot be freely estimated; therefore, no initial residual variances were set (Muthen, 1990). Factors and residual variances were independent of each other. Note that the correlation between two within-level factors was also 0.30 because within-level factor variances were fixed at 1.00. The threshold of indicators was equal to 0, resulting in a 50:50 proportion of responses that are 0 or 1.

The between-level model had an identical factorial structure to the within-level model. Parameters for simulation data generation were factor loadings = 0.70, residual variances = 0.51. Factors and residuals were independent of each other. We varied the variance of between-level factors (a in Fig. 1) to create three ICC conditions. (More detailed information is presented in the next section.) Note that for each ICC condition, we adjusted the between-level factor covariance (b in Fig. 1) based on the formula: 0.30 $\times \sqrt{\mathit{\operatorname{var}}\left({\eta}_{B1}\right)}\times \sqrt{\mathit{\operatorname{var}}\left({\eta}_{B2}\right)}$ so that the correlation of two between-level factors can be held to 0.30 across different ICC conditions.

Design factors

Numbers of groups (NG)

A NG larger than 100 was recommended as an acceptable estimate at the between-level with low ICC (Hox & Maas, 2001; Hsu et al., 2015). Ryu and West (2009) used NG = 50, 100, 200, and 1000 in their simulation work, where NG = 50 could lead to nonconvergence problems and NG = 1000 was not a realistic NC for practitioners. Hsu et al. (2017) found that NG = 200 seemed to compensate for convergence problems if ICC was low. As a result, this study considered NG ranging from 50 to 200. Specifically, the current study adopted three NG conditions (50, 100, and 200) to evaluate the impact of NC on the performance of level-specific fit indices when the indicators of MCFA were dichotomous.

Group size (GS)

Hox and Maas (2001) used a set of GS = 10, 20, 50 on a MSEM study and found that GS had a trivial impact on parameter estimates, as well as standard errors. Additionally, Ryu and West (2009) used GS = 20, 50, and 100 in their study focusing on performance of fit indices of MSEM. In consideration of common practices and comparability, the present study adopted GS = 10, 20, and 50.

ICC. The ICC (ρ) is defined as:

$$\rho =\frac{VAR_{Between}}{VAR_{Between}+{VAR}_{Within}}$$

(6)

where VAR_Between and VAR_Within are the variance of between-level factors and within-level factors, respectively. The variance of within-level factors was constrained to 1.00, while that of between-level factors varied to create different ICC conditions. The present study considered three levels of between-level variance (0.1, 0.3, and 0.5), resulting in three levels of ICC: .091 (ICC1), .231 (ICC2), and .333 (ICC3).

Thresholds of dichotomous indicators (THR)

To the best of our knowledge, study of the impact of asymmetric thresholds on l-s fit indices is still lacking. Guided by Rhemtulla et al.’s (2012) and Savalei and Rhemtulla’s (2013) studies, we considered both symmetric (50:50) and asymmetric (80:20) conditions. By considering these design factors, we intended to understand the impact of asymmetry of categorical data on the performance of l-s χ² and fit indices.

Factor loadings (FL)

Following Nestler (2013) simulation study, we adopted three conditions of FL in this study: low (.40), medium (.55), and high (.70). The setting of FL was also in line with previous simulation studies (Forero et al., 2009; Garrido et al., 2016; Heene et al., 2011). Previous simulation studies have shown that low magnitudes of factor loadings resulted in less sensitivity of traditional fit indices. By including this design factor, we intended to examine the extent to which the magnitudes of factor loadings can impact the performance of l-s χ² and fit indices.

As a result, a total of 162 conditions (NG = 50, 100, and 200; GS = 10, 20, and 50; ICC = ICC1 to ICC3; THR = 50:50 and 80:20; FL = .40, .55, and .70) were yielded. For each condition, 500 replications were generated using Mplus 7.4 (Muthen & Muthen, 1998–2015).

Intentional misspecifications in the hypothesized models

After simulation data were generated, we analyzed simulation data in three different conditions: (a) correctly specified hypothesized model (i.e., the hypothesized model was equal to the population model as shown in Fig. 1, M_C); (b) misspecification in between-level model only (M_B, see Fig. 2); and (c) misspecification in within-level model only (M_W, see Fig. 2). Following Ryu and West’s (2009) and Hsu et al.’s (2017) studies, M_B contained a misspecification where only one between-level factor loaded on the indicators. M_W contained a misspecification where only one within-level factor loaded on the indicators. The WLSMV estimator was applied in three conditions to obtain the model solutions in Mplus. Starting values of parameter estimates were set to the same as the parameter values in the population model to prevent any convergence problems due to bad starting values. Fit indices of interest were computed in each condition.

Fig. 2

a Misspecification in the between-level model only (M_B). b Misspecification in the within-level model only (M_W)

Analysis

In M_C, M_B, and M_W conditions, the values of fit indices were saved for subsequent analyses. Deceptive statistics for level-specific χ² test statistics and fit indices were computed and reported. If needed, factorial analysis of variance (ANOVA) was applied to determine the impact of the design factors on the performance of fit indices (Skrondal, 2000). Specifically, eta-squared (η²) was reported to indicate the proportion of the variance accounted for by a particular design factor or the interaction effect terms. Following Cohen’s (1988, 1992) suggestion, we used a moderate η² above .06 (i.e., practically significant) to identify influential design factors in the fit indices values. Note that when a fit index had a standard deviation close to 0, the impact of design factors on the values of the fit index were self-evidently trivial, regardless of the η² values. In addition, for level-specific χ² test statistics, we computed the type I error rates (under M_C) or statistical power (under M_B and M_W) under α level = .05. For level-specific fit indices, we applied the traditional cut-off values (RMSEA-related fit indices < .06; CFI- and TLI-related fit indices > .95; SRMR-related fit indices < .08; Hu & Bentler, 1999) to explore whether these fit indices were promising to indicate correctly specified or misspecified hypothesized models. Note that our intention was not to encourage using traditional cut-off values as golden rules for model evaluation. Rather, we intended to provide a sense of whether these cut-off values are applicable when level-specific χ² test statistics or fit indices are used.

Results

Convergence rates

The convergence rates were highly associated with magnitude of ICC and FL, followed by sample size (NG and GS). In contract, THR was less influential. In this section, the convergence rates were reported by ICC and FL. First, the convergence rates were above 95% in conditions with high ICC (.333) and high FL (.70), with one exception (the convergence rate was 92% when NG = 50, GS = 10, and THR = 80:20). In conditions with high ICC and medium FL (.55), the coverage rates were close to or above 95% only when NG/GS were at least 100/20 or NG/GS were at least 200/10. On the other hand, in conditions with high ICC and low FL (.40), the coverage rates were below 95% (ranging between 46% and 89%). Second, in conditions with medium ICC (.231) and high FL, the convergence rates were above 95% when NG/GS were at least 100/10 and THR was symmetric. In contrast, in conditions with medium ICC and medium FL, the convergence rates were above 95% only when NG/GS were at least 200/50. Unfortunately, in conditions with medium ICC and low FL, the coverage rates were below 95% (ranging between 37% and 63%). Third, in conditions with low ICC (.091), the coverage rates were below 95% – in range of 48% and 84% when FL was high; in range of 40% and 52% when FL was medium; and in range of 29% and 47% when FL was low. Only converged solutions were used for further analysis.

Effects of design factors on level-specific fit indices

The left-hand side of Table 1 summarizes the descriptive statistics [aggregated means and standard deviations (SDs)] of level-specific χ² test statistics and fit indices for the M_C, M_B, and M_W conditions. The right-hand side of Table 1. shows the η² values derived from ANOVA results using level-specific χ² test statistics and fit indices as outcomes. We have highlighted η² values above .06 (Cohen, 1988, 1992) in grey to show practically significant effects of design factors. Note that when a level-specific χ² test statistic or fit index has a small variation (indicated by a small SD), the impact of design factors was self-evidently trivial even if any η² exceeding .06 is identified. Note the η² values of three-way interactions were close to 0, and therefore, were not reported in Table 1 for the sake of simplicity. In addition, to inform the performance of level-specific χ² test statistics and fit indices, we report the average values of level-specific χ² test statistics and fit indices by all design factors in Appendix B (Tables 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 and 21), where type I error rates or statistical power are reported for level-specific χ² test statistics.

Table 1 ANOVA results (η²) with χ² test statistics and fit indices values as the dependent variables for the * NG * GS * ICC * THR * FL design

In the following section, we reported the simulation of between-level-specific and within-level-specific χ² and fit indices separately. To better inform interested readers of our findings, we created Tables 2 and 3 to summarize the simulation results for between-level-specific and within-level-specific χ² and fit indices, respectively. In addition, our simulation results suggest the performance of between-level-specific χ² and fit indices was complicated and more likely influenced by NG, ICC, and FL. We visualized the values of between-level-specific fit indices in Figs. 3 and 4 (by NG, ICC, and FL with GS set to 50.) for M_C and M_B conditions, respectively. Also, because we found THR could slightly weigh in the performance of between-level-specific fit indices in some conditions, the visualization was presented for symmetric THR and asymmetric THR conditions separately. Note we did not present the visualization of within-level-specific fit indices because the patterns of their performance were less complicated and could be clearly described in Table 3.

Table 2 Summary of simulation findings for between-level-specific χ² and fit indices

Table 3 Summary of simulation findings for within-level-specific χ² and fit indices

Fig. 3

Values of between-level-specific fit indices under M_C. Note. M_C = correctly specified hypothesized model. NG = numbers of groups. GS = group size. ICC = intraclass correlation coefficient. THR = threshold of dichotomous indicators (symmetric = 50:50 and asymmetric = 80:20). FL = factor loadings.

Note. M_C = correctly specified hypothesized model. NG = numbers of groups. GS = group size. ICC = intraclass correlation coefficient. THR = threshold of dichotomous indicators (symmetric = 50:50 and asymmetric = 80:20). FL = factor loadings

Fig. 4

Values of between-level-specific fit indices under M_B. Note. M_B = misspecification in between-level model only. NG = numbers of groups. GS = group size. ICC = intraclass correlation coefficient. THR = threshold of dichotomous indicators (symmetric = 50:50 and asymmetric = 80:20). FL = factor loadings

Between-level-specific χ ² and fit indices

${\boldsymbol{\chi}}_{\boldsymbol{B}}^{\textbf{2}}$ . As shown in Table 1, under M_C where the hypothesized between-level model was correctly specified, ${\chi}_B^2$ had a mean of 31.53, an SD of 6.45, and the η²s ranged from .00 to .03. Small values of η²s suggest that five design factors (NG, GS, ICC, THR, and FL) accounted for trivial variance of ${\chi}_B^2$. That is, our design factors had a negligible effect on ${\chi}_B^2$. Appendix B shows that the type I error rates of ${\chi}_B^2$ were generally acceptable (close to or below .05) across all conditions. On the other hand, under M_B, where the hypothesized between-level model was misspecified, ${\chi}_B^2$ had a mean of 45.22 and an SD of 23.77 (see Table 1). FL (η² = .10), ICC (η² = .09), NG (η² = .06), and ICC*FL (η² = .07) can jointly affect ${\chi}_B^2$. Based on data in Appendix B, we found ${\chi}_B^2$ had higher statistical power to detect misspecified between-level models when FL, ICC, or NG increased. These results show that when ${\chi}_B^2$ was used, type I error rates might not be a concern, but statistical power would be of problem when FL, ICC, or NG were low.

We further explore the statistical power of ${\chi}_B^2$ reported in Appendix B. First, we found statistical power was far below .80 under several conditions: NG = 50 or GS = 10, or ICC1 or low FL. Second, we found ${\chi}_B^2$ was able to reach satisfying statistical power (close to or above .80) under ICC3 with (a) high FL and NG = 100, or (b) medium FL and NG = 200. If ICC was medium (ICC2), only the following condition resulted in satisfying statistical power: high FL, NG/GS at least 200/20, and symmetric THR. The findings regarding the performance of ${\chi}_B^2$ were summarized in Table 2.

RMSEA_B

Table 1 shows that under M_C, RMSEA_B had a close-to-zero mean (0.01) and SD (0.02), suggesting that RMSEA_B had a desired characteristic of indicating good model fit across replications when the between-level model was correctly specified. The η²s of RMSEA_B were below .02, indicating that the design factors had a trivial impact on RMSEA_B. Figure 3 and Appendix B show that RMSEA_B had means below the traditional cut-off value of .06 (Hu & Bentler, 1999) in all conditions. Alternatively, under M_B, as shown in Table 1, RMSEA_B had a mean of 0.03 and an SD of 0.04. ICC (η² = .14) and FL (η² = .14) can dominantly account for the variance of RMSEA_B, followed by ICC*FL (η² = .08). According to Fig. 4 and Appendix B, we found RMSEA_B was more likely to identify misspecified hypothesized between-level models when ICC or FL increased. In addition, the impact of ICC on RMSEA_B was small when FL was low, and the impact of ICC became greater when FL increased. These results show that low magnitudes of ICC or FL in models might interfere with the capacity of RMSEA_B to correctly determine model fit.

We dug into tables in Appendix B to determine the extent to which RMSEA_B could be used to determine model fit when the cut-off value of .06 was applied. We found RMSEA_B cannot be used to determine model fit under ICC1 or low FL conditions. On the other hand, RMSEA_B could be widely used in conditions with high ICC (ICC3) and high FL. In conditions with ICC3 and medium FL, increasing NG to at least 200 could prevent RMSEA_B from under-rejecting poor model fit. The performance of RMSEA_B is summarized in Table 2.

CFI _B and TLI _B

Under M_C, CFI_B and TLI_B were expected to be above the traditional cut-off value of .95 (Hu & Bentler, 1999). However, as reported in Table 1, CFI_B and TLI_B had means (0.81 and 0.79, respectively) far below .95 and the SDs (0.35 and 0.36, respectively) were not negligible. This result suggested that CFI_B and TLI_B were not necessarily promising to indicate correctly hypothesized between-level models across replications. Their η²s provided more information about when these two fit indices could be useful. Results show our design factors influenced CFI_B and TLI_B in a similar way. Particularly, NG (η² = .08 and .07 for CFI_B and TLI_B, respectively), ICC (η² = .09 for CFI_B and TLI_B), and FL (η² = .10 and .09 for CFI_B and TLI_B, respectively) had a similar magnitude of impacts on CFI_B and TLI_B. Based on Appendix B and Fig. 3, we found that both fit indices were more promising to identify correctly hypothesized between-level models when FL, ICC, or NG increased.

Under M_B, CFI_B and TLI_B also acted similarly. As presented in Table 1, CFI_B and TLI_B had means (0.65 and 0.59, respectively) far below the traditional cut-off value of .95, which were satisfying, with SDs equal to 0.34 and 0.36, respectively. The values of η²s ranged from .00 to .04, suggesting that design factors had minimal effects on CFI_B and TLI_B. Taken together, the results derived from M_C and M_B conditions show that low magnitudes of NG, ICC, or FL in models might limit the performance of CFI_B and TLI_B in correctly determining model fit.

We then examined tables in Appendix B to understand the performance of CFI_B and TLI_B under different conditions when the cut-off value of .95 was applied. First, we found that CFI_B and TLI_B cannot be used to determine model fit under ICC1 or low FL conditions. Second, if either ICC or FL were medium, increasing NG to at least 200 could prevent CFI_B and TLI_B from over-rejecting good model fit. Third, both indices could be widely used in conditions with ICC3, high FL, and NG at least 100. We summarized the performance of CFI_B and TLI_B in detail in Table 2.

SRMR_B

Under M_C, SRMR_B had a mean of 0.09, which was greater than a traditional cut-off value of .08; (Hu & Bentler, 1999), and an SD of 0.03 (see Table 1), suggesting that SRMR_B might not be able to correctly indicate good model fit in all conditions. NG (η² = .56) accounted for a substantial proportion of SRMR_B’s variances followed by GS (η² = .09). Figure 3 and Appendix B shows that SRMR_B was more promising to identify correctly hypothesized between-level models when NG or GS increased.

On the other hand, under M_B, SRMR_B had a mean of 0.10 and an SD of 0.03 (see Table 1.) and it can be impacted by NG (η² = .42), FL (η² = .08), and GS (η² = .07). Based on Appendix B, we found SRMR_B was more promising when FL increased. However, we also found SRMR_B was less promising to correctly indicate poor models when NG or GS increased. After we took a close look at Figs. 3 and 4 as well as tables in Appendix B, we found this result was not surprising and could result from the fact that the patterns/values of SRMR_B found under M_C were reproduced under M_B in many conditions, including low NG, low FL, ICC1, or ICC2 with medium FL [note ICC and ICC*FL Table 1 (η² = .03, respectively) in variance of SRMR_B]. SRMR_B performance under M_B should be interpreted cautiously. Put together, one should be aware that low magnitude of NG, FL, or ICC might cause SRMR_B to lose the capacity to correctly determine model fit.

Using the tables within Appendix B, we found some conditions determining whether SRMR_B could be useful when the cut-off value of .08 was applied. First, as aforementioned, we found SRMR_B cannot be used to determine model fit under (a) low NG, (b) low FL, (c) ICC1, or (d) ICC2 with medium FL conditions. Second, in conditions with (a) ICC3 and high/medium FL or (b) ICC2 with high FL, SRMR_B was useful if NG/GS were at least 100/50 or NG/GS were at least 200/10. When asymmetric THR appeared, NG/GS of at least 200/10 could be considered. We summarized the performance of SRMR_B in Table 2.

Within-level-specific χ ² and fit indices

${\boldsymbol{\chi}}_{\boldsymbol{W}}^{\textbf{2}}$ . As presented in Table 1, under M_C, ${\chi}_B^2$ had a mean of 31.84, an SD of 7.14, and η²s ranging from .00 to .09. GS was the only influential design factor (η² = .09); however, Appendix B shows that type I error rates of ${\chi}_W^2$ were close to or below .05 across all conditions. That is, GS did not significantly affect the capacity of ${\chi}_W^2$ to identify correctly hypothesized within-level models. In contrast, under M_W, ${\chi}_W^2$ had a mean of 306.75 and an SD of 377.37, as shown in Table 1. Large variations in values of ${\chi}_W^2$ resulted from FL (η² = .14), NG (η² = .11), GS (η² = .08), NG*FL (η² = .06), and GS*FL (η² = .06). Based on Appendix B, we found ${\chi}_W^2$ had higher statistical power (close to or above .80) to detect misspecified within-level models when NG, GS, and FL increased. The above results show when using ${\chi}_W^2$, one should be aware that type I error rates might not be a problem, but low statistical power might appear when NG, GS, or FL were low.

We examined Appendix B to uncover the conditions where ${\chi}_W^2$ had statistical power close to or above .80. First, when FL was high, ${\chi}_W^2$ was able to reach satisfying statistical power given our minimum combination of NG/GS (50/10). Second, when FL was medium, thresholds of dichotomous indicators started to slightly weigh into the requirement of NG and GS (THR had a η² of .03, as shown in Table 1): NG/GS = 50/10 was sufficient for symmetric THR, while NG/GS of at least 50/20 or at least 100/10 was required for asymmetric THR. Third, when FL was low, THR could weigh in a little more: NG/GS of at least 100/20 or 200/10 was sufficient for symmetric THR; NG/GS of at least 100/50 or 200/20 for asymmetric THR. The performance of ${\chi}_W^2$ is summarized in Table 3.

RMSEA_W

Table 1 shows that under M_C, RMSEA_W had a mean equal to 0.00 and an SD equal to 0.01, suggesting that RMSEA_W shows a desired property of indicating good model fit across replications when the within-level model was correctly specified. GS (η² = .10) was the only factor that significantly influenced RMSEA_W. Nevertheless, the impact of GS on RMSEA_W could be ignored because the variation of RMSEA_W was close to zero; thus the impact of GS was self-evidently trivial. RMSEA_W had means below a traditional cut-off value of .06 (Hu & Bentler, 1999) in all conditions.

On the other hand, under M_B, as shown in Table 1, RMSEA_W had a mean of 0.04, an SD of 0.02. FL (η² = .56) dominantly affected RMSEA_W, followed by THR (η² = .09). The tables in Appendix B show that RMSEA_W was more likely to identify misspecified hypothesized within-level models when FL was increased or THR was symmetric. After examining tables in Appendix B, we found RMSEA_W tended to under-reject poor model fit when the cut-off value of .06 was applied. Particularly, results show RMSEA_W can be used to determine model fit in very restricted conditions: when FL was high and THR was symmetric. The performance of RMSEA_W is summarized in Table 3.

CFI _W and TLI _W

In general, CFI_W and TLI_W performed similarly. As presented in Table 1, under M_C, both CFI_W and TLI_W had a mean equal to 0.99 and an SD equal to 0.03, suggesting that these two fit indices were able to correctly indicate good model fit across replications. The η² of GS was .07 for CFI_W and .08 for TLI_W; however, the impact of GS could be neglected due to a small variation in CFI_W and TLI_W. Under M_w, CFI_W and TLI_W had means of 0.74 and 0.66, and SDs of 0.08 and 0.10, respectively. Although FL seemed to affect CFI_W and TLI_W (η² = .09), Appendix B reveals that both CFI_W and TLI_W had means far below a traditional cut-off value of .95 (Hu & Bentler, 1999) in most conditions, suggesting that these two fit indices were promising to identify incorrectly hypothesized within-level models.

We examined Appendix B to understand the sensitivity of CFI_W and TLI_W under different conditions when the cut-off value of .95 was applied. We found that CFI_W and TLI_W can be widely used with the cut-off value of .95 except for the conditions with low FL, NG/GS = 50/10, and asymmetric THR. We summarized the performance of CFI_W and TLI_W in Table 3.

SRMR_W

Under M_C, SRMR_W had a mean of 0.04, which was below a traditional cut-off value of .08 (Hu & Bentler, 1999), and a small SD of 0.02 (shown in Table 1). The results suggest that SRMR_W showed promise to correctly indicate good model fit in most replications. NG (η² = .29), GS (η² = .40), and THR (η² = .08), were practically significant but could be ignored due to a small variation in SRMR_W. On the other hand, under M_B, SRMR_W had a mean of 0.09, which was above a traditional cut-off value of .08, and a small SD of 0.02, suggesting that SRMR_W was useful to indicate poor model fit in most replications. The influence of FL (η² = .54) and GS (η² = .06) could be ignored since the variation in SRMR_W was trivial. Using Appendix B, we found SRMR_W with a cut-off value of .08 can be used to determine model fit in all conditions except for the conditions with low FL. We summarized the performance of SRMR_W in Table 3.

Discussion

The current study examined the performance of level-specific χ² test statistics and fit indices derived from the partially saturated-model method in correctly identifying good or poor model fit in MCFA with dichotomous indicators. Tables 2 and 3 summarize our simulation results that could inform applied researchers of their usage of level-specific χ² test statistics and fit indices in evaluating MCFA with dichotomous indicators. Note that the results were tied to the specific conditions of this research and should not be overgeneralized. In this section, we highlight major findings for discussion.

First, as summarized in Table 2, the performance of b-l-s χ² and fit indices was highly associated with the magnitude of ICC in data. Both b-l-s χ² and fit indices were more promising indicators to correctly indicate model fit in MCFA with dichotomous indicators when ICC increased. Unfortunately, in conditions with low ICC (ICC1 = .091), we found ${\chi}_B^2$ had statistical power far below .80, and all b-l-s fit indices (RMSEA_B, CFI_B, TLI_B, and SRMR_B) were not useful to evaluate between-level models along with traditional cut-off values. This is congruent with previous findings derived from MCFA with continuous indicators (Hsu et al., 2017; Lee & Sohn, 2022). Note that between-level indicators have stronger relations when ICC is higher, allowing for possibly “greater” discrepancies between the model-implied and observed between-level variance-covariance matrices (Kline, 2011). As a result, given identical misspecification and sample size, the value of ${\chi}_B^2$ increased (higher statistical power to detect the misspecification) with higher ICC. Because RMSEA_B is a function of the ${\chi}_B^2$, the pattern of RMSEA_B in response to ICC was similar to ${\chi}_B^2$. Regarding CFI_B and TLI_B, both were functions of the distance between the χ² value of the hypothesized model and that of the baseline null model—the greater the distance between two χ² values, the larger were CFI_B and TLI_B. When the between-level model was misspecified, a higher ICC not only led to greater χ² values of the hypothesized model and the baseline null model, but also resulted in larger distances between these two χ² values. That is why we observed that both CFI_B and TLI_B were more promising for detecting misspecified between-level models when ICC increased. Similarly, SRMR_B was more effective when ICC was higher. This is because SRMR_B reflects the deviation between the between-level model-implied and observed variance-covariance matrices; the deviation would become larger with an increase in ICC. On the other hand, we found that ICC did not affect the effectiveness of w-l-s χ² and fit indices because the within-level variation was held constant across the ICC conditions in this study (i.e., within-level indicators had constant relations across ICC conditions).

Second, as presented in Table 2, another factor influencing the performance of b-l-s χ² and fit indices was the magnitude of factor loadings. Decreasing factor loadings were associated with decreasing value of ${\chi}_B^2$ and of all b-l-s fit indices. In conditions with low FL (.40), the statistical power of ${\chi}_B^2$ was far below .80, and no b-l-s fit indices can correctly identify model fit along with traditional cut-off values. These findings are in line with Nestler’s (2013) and Heene et al.’s (2011) studies. As noted by Heene et al. (2011), the problem of using ${\chi}_B^2$, RMSEA_B, and SRMR_B in low FL conditions is that the values of these indicators will be too small to detect the misspecified between-level models. On the other hand, the problem of using CFI_B and TLI_B in low FL conditions is that these indicators will over-reject correctly hypothesized between-level models.

Third, in addition to ICC and FL, the size of NG played an important role in the performance of ${\chi}_B^2$ as well as b-l-s fit indices if traditional cut-off values were applied. As shown in Table 2, a small to medium NG (50–100) might be sufficient for b-l-s χ² and fit indices only if both ICC and FL were high. In the remaining conditions, an NG of 200 was needed to use b-l-s χ² and fit indices for between-level model evaluation. These findings raise a pressing issue—when both ICC and FL are not high, how can we assess the between-level model fit in MCFA with dichotomous indicators given an NG far less than 200? Thus far, the possible solution we are aware of is to compute adjusted ${\chi}_B^2$ test statistics based on the post hoc corrections (Herzog & Boomsma, 2009; Savalei, 2010). Specifically, when the NG is small (i.e., small sample size at the between-level), the estimated ${\chi}_B^2$ might not follow χ² distribution (Bentler & Yuan, 1999). Previous research has shown that computing adjusted χ² is promising in providing sufficient statistical power and reliable model fit for data with nonnormality or small sample size (Savalei, 2010). To the best of our knowledge, the effectiveness of adjusted ${\chi}_B^2$ test statistics based on the post hoc corrections in small NG scenarios has not been studied in MCFA; future studies are needed to shed new light on this issue. In addition, little is known about whether b-l-s fit indices computed based on adjusted ${\chi}_B^2$ are useful for detecting misspecified between-level models in MCFA. Future studies are encouraged to explore this topic.

Fourth, as shown in Table 1, THR had close-to-zero η²s for b-l-s χ² and fit indices, suggesting the impact of asymmetric thresholds (80:20) on the performance of b-l-s χ² and fit indices might be trivial. Still, we did discover that the impact of THR can slightly weigh into the performance of b-l-s χ² and fit indices when ICC was not high. For example, as summarized in Table 2, when ICC was medium (ICC2), we can reach a satisfying statistical power of ${\chi}_B^2$ only in conditions with high FL, large NG (200), and symmetric THR, but not in any conditions with asymmetric THR. This finding is in line with Rhemtulla et al. (2012) and Savalei and Rhemtulla (2013), who found that asymmetric thresholds and small samples reduced adjusted χ² statistical power. In addition, we observed that under ICC2, b-l-s fit indices might need larger NG/GS to compensate for the presence of asymmetric thresholds. For instance, as shown in Table 2, in conditions with ICC2 and high FL, CFI_B and TLI_B required NG/GS of at least 100/50 or NG/GS of at least 200/10 if thresholds were symmetric, but CFI_B and TLI_B would require NG/GS at least 200/50 if thresholds were asymmetric. This finding might inform applied researchers of the sample size requirement when asymmetric thresholds occur.

Fifth, the magnitude of factor loadings and asymmetric thresholds should be considered when using w-l-s χ² and fit indices. As presented in Table 3, w-l-s χ² and fit indices except for RMSEA_B were useful in most conditions to determine model fit at within-level in MCFA with dichotomous indicators. Our study complements earlier work (Hsu, 2009; Hsu et al., 2017; Ryu & West, 2009) that found that w-l-s χ² and fit indices performed well in MCFA with continuous indicators, and our findings further support the adequacy of adopting w-l-s χ² and fit indices to evaluate within-level models in MCFA if indicators are dichotomous. However, we found that both FL and THR could influence the performance of w-l-s χ² and fit indices. As aforementioned, decreasing factor loadings were associated with decreasing value of χ² and of fit indices. As a result, one would need to increase NG/GS to compensate for the impact of low factor loadings. In addition, we found THR had a slight impact and could weigh in the performance of ${\chi}_W^2$, RMSEA_W, CFI_W, and TLI_W. Table 3 suggested RMSEA_W was heavily affected by FL and THR: RMSEA_W can be used to determine model fit only when FL was high and THR was symmetric. The good news is that others fit indices were useful for determining within-level model fit along with traditional cut-off values.

Last but not least, the results suggest that a low ICC (.091) was linked to the low convergence rates in MCFA with dichotomous indicators. This finding was in line with Hox and Maas’s (2001) and Hsu et al.’s (2017) study findings that were based on an MCFA population model with continuous indicators. Due to the fact that ICCs around .10 seem to be very common in applied research. Explore alternative estimation strategies (e.g., Bayesian estimation; see Depaoli & Clifton, 2015) to resolve the issue (i.e., low convergence rates) for multilevel confirmatory factor analysis can be crucial. In other words, stable and reliable estimation methods need to be developed and evaluated before addressing further questions about model fit.

Several limitations should be noted. First, our findings should be generalized only to studies that apply MCFA models with dichotomous indicators (Fig. 1). Further studies are needed to test the replicability of our results using different models (e.g., structural models). Second, only a limited number of design factors were considered in the current study. Future studies could expand the examination of additional design factors, such as unequal factor loadings, unequal group conditions, the number of observed indicators per latent factor, and different types of misspecifications (e.g., misspecified in cross-loadings; Moshagen & Auerswald, 2018). Third, considering the computational complexity in MCFA with dichotomous indicators, we generated 500 replications for each of 162 conditions and investigated one misspecification condition in this study which restricted the ability to generalize. Future research could extend our investigation using a more powerful computer.

Concluding remarks

According to our simulation results, we recommend that practitioners should be aware that the performance of b-l-s χ² and fit indices was mainly influenced by data ICC and factor loadings (FL), followed by number of groups (NG), while thresholds of dichotomous indicators (THR) could slightly weigh in the performance of b-l-s fit indices in some conditions. Specifically, both b-l-s χ² and fit indices were more promising indicators to correctly indicate model fit in MCFA with dichotomous indicators when ICC or FL increased. In conditions with low ICC (.091) or low FL (.40), ${\chi}_B^2$ had statistical power far below .80, and all b-l-s fit indices were not useful to evaluate between-level models along with traditional cut-off values. In terms of NG, we found a small to medium NG (50–100) might be sufficient for b-l-s χ² and fit indices only if both ICC and factor loadings were high, while in remaining conditions, an NG of 200 was needed to be able to use b-l-s χ² and fit indices for between-level model evaluation. The performance of b-l-s χ² and fit indices is summarized in Table 2. On the other hand, we recommend that practitioners use w-l-s χ² and fit indices (except for RMSEA_W) along with traditional cut-off values to evaluate within-level models comprising dichotomous indicators. However, practitioners should be aware that both FL and THR could influence the performance of w-l-s χ² and fit indices. Both w-l-s χ² and of fit indices were more promising to determine model fit when FL increased. When FL was low, one would need to increase NG and group size (GS) to compensate for the impact of low factor loadings. THR had a slight impact and could weigh in the performance of ${\chi}_W^2$, RMSEA_W, CFI_W, and TLI_W. Unfortunately, RMSEA_W was heavily affected by FL and THR: RMSEA_W can be used to determine model fit only when FL was high and THR was symmetric. The performance of w-l-s χ² and fit indices is summarized in Table 3.

Appendices

Appendix A. The formulas for the level-specific fit indexes

Chi-square statistic

By applying the PS method, the ${\chi}_{PS\_B}^2$ can be defined as follows:

$${\chi^2}_{PS\_B}={F}_{ML}\left[{\varSigma}_B\left(\hat{\theta}\right),{\varSigma}_W\left({\hat{\theta}}_S\right)\right]-{F}_{ML}\left[{\varSigma}_B\left({\hat{\theta}}_S\right),{\varSigma}_W\left({\hat{\theta}}_S\right)\right]$$

(A1)

in which the ${F}_{ML}\left[{\varSigma}_B\left(\hat{\theta}\right),{\varSigma}_W\left({\hat{\theta}}_S\right)\right]$ is the fitting function for the saturated within-level model, while the ${F}_{ML}\left[{\varSigma}_B\left({\hat{\theta}}_S\right),{\varSigma}_W\left({\hat{\theta}}_S\right)\right]$ is the fitting function when both within-level and between-level levels are saturated, i.e., the fully saturated model. The degrees of freedom of ${\chi}_{PS\_B}^2$, denoted by df_{PS _ B}, equals to the difference between the number of parameters in the partially saturated model and the fully saturated model and can be defined in terms of A2 as follows:

$${df}_{PS\_B}={df}_{B,\, Saturated}-{df}_{B,\, Hypothesized}$$

(A2)

in which df_{B, Saturated} and df_{B, Hypothesized} represent number of parameters in the saturated within-level model and fully saturated model, respectively.

Similarly, the ${\chi}_{PS\_W}^2$ and its corresponding degree of freedom df_{PS _ W} can be derived in terms of A3 and A4, respectively, as follows:

$${\upchi^2}_{\textrm{PS}\_\mathrm{W}}={\textrm{F}}_{\mathrm{ML}}\left[{\Sigma}_{\textrm{B}}\left({\hat{\uptheta}}_{\textrm{S}}\right),{\Sigma}_{\textrm{W}}\left(\hat{\uptheta}\right)\right]-{\textrm{F}}_{\mathrm{ML}}\left[{\Sigma}_{\textrm{B}}\left({\hat{\uptheta}}_{\textrm{S}}\right),{\Sigma}_{\textrm{W}}\left({\hat{\uptheta}}_{\textrm{S}}\right)\right]$$

(A3)

$${\textrm{df}}_{\mathrm{PS}\_\textrm{W}}={\mathrm{df}}_{\textrm{W},\,\mathrm{Saturated}}-{\textrm{df}}_{\mathrm{W},\,\textrm{Hypothesized}}$$

(A4)

RMSEA

Given ${\chi}_{PS\_B}^2$ and df_{PS _ B} , the RMSEA for the between-level model, RMSEA_{PS_B}, can be derived in terms of A5 as follows:

$${RMSEA}_{PS\_B}=\sqrt{\operatorname{Max}\ \left(\frac{\chi_{PS\_B}^2-{df}_{PS\_B}}{df_{PS\_B}(J)},0\right)}$$

(A5)

The J in A5 is the sample size (number of groups) for the between-level model. To reduce the random error component of F_ML, the non-central parameter df_{PS _ B} is subtracted from the ${\chi}_{PS\_B}^2$ (Rigdon, 1996). Therefore, ${\chi}_{PS\_B}^2-{df}_{PS\_B}$ stands for an unbiased estimator of F_ML. The denominator J in A5 functions as a penalty for lack of parsimony. The df_{PS _ B} in the denominator transfers $\left({\chi}_{PS\_B}^2-{df}_{PS\_B}\right)/\textrm{J}$ into a measure of lack of fit per df. In addition, the RMSEA_{PS _ B} is is set to zero providing ${\chi}_{PS\_B}^2$ is smaller than df_{PS _ B}. Similarly, the RMSEA_{PS_W} can be derived from A6, in which N denotes the total sample size for the within-level model:

$${RMSEA}_{PS\_W}=\sqrt{\operatorname{Max}\ \left(\frac{\upchi_{\textrm{PS}\_\mathrm{W}}^2-{\textrm{df}}_{\mathrm{PS}\_\textrm{W}}}{{\mathrm{df}}_{\textrm{PS}\_\mathrm{W}}\left(\textrm{N}-\mathrm{J}\right)},0\right)}$$

(A6)

CFI

CFI is one incremental fit index that can be used to evaluate the goodness-of-fit by comparing the hypothesized model with the independence model (Bentler, 1990). The CFI for the between-level model (CFI_{PS _ B}), derived from those proposed by Ryu and West (2009), is showed as follows:

$${CFI}_{PS\_B}=1-\frac{\operatorname{Max}\left[\ \left({\upchi}_{\textrm{PS}\_\mathrm{B}}^2-{\textrm{df}}_{\mathrm{PS}\_\textrm{B}}\right),0\right]}{\operatorname{Max}\left[\ \left({\upchi^2}_{\textrm{I}\_\mathrm{B},\textrm{S}\_\mathrm{W}}-{\textrm{df}}_{\mathrm{I}\_\textrm{B},\mathrm{S}\_\textrm{W}}\right),0\right]}$$

(A7)

in which χ²_{I _ B, S _ W} represents the χ² test statistic with an independence (null) between-level model and a saturated within-level model. The χ²_{I _ B, S _ W} can be derived in term of A8 as follows:

$${\chi^2}_{\textrm{I}\_\mathrm{B},\textrm{S}\_\mathrm{W}}={\textrm{F}}_{\mathrm{ML}}\left[{\Sigma}_{\textrm{B}}\left({\hat{\uptheta}}_{\textrm{I}}\right),{\Sigma}_{\textrm{W}}\left({\hat{\uptheta}}_{\textrm{S}}\right)\right]-{\textrm{F}}_{\mathrm{ML}}\left[{\Sigma}_{\textrm{B}}\left({\hat{\uptheta}}_{\textrm{S}}\right),{\Sigma}_{\textrm{W}}\left({\hat{\uptheta}}_{\textrm{S}}\right)\right]$$

(A8)

The independence model aforementioned can be derived by correlating all the observed variables and constraining all the correlation coefficients to zero. The corresponding df_{I _ B, S _ W} is defined as:

$${df}_{I\_B,S\_W}={df}_{B,\, Saturated}-{df}_{B,\, Independent}$$

(A9)

The CFI_{PS _ B} measures the goodness-of-fit by comparing the hypothesized model with the independence model at the between level, when the within-level model is saturated. Similarly, the CFI_{PS _ W} can be derived in terms of A10 as follows:

$${CFI}_{PS\_W}=1-\frac{\operatorname{Max}\left[\ \left({\chi}_{PS\_W}^2-{df}_{PS\_W}\right),0\right]}{\operatorname{Max}\left[\ \left({\chi^2}_{S\_B,I\_W}-{df}_{S\_B,I\_W}\right),0\right]}$$

(A10)

where χ²_{S _ B, I _ W} showed in A11 represents the χ² statistics with a saturated between-level model and an independence within-level model. The corresponding df_{S _ B, I _ W} can be derived in terms of A12 as follows:

$${\chi^2}_{S\_B,I\_W}={\textrm{F}}_{\mathrm{ML}}\left[{\Sigma}_{\textrm{B}}\left({\hat{\uptheta}}_{\textrm{S}}\right),{\Sigma}_{\textrm{W}}\left({\hat{\uptheta}}_{\textrm{I}}\right)\right]-{\textrm{F}}_{\mathrm{ML}}\left[{\Sigma}_{\textrm{B}}\left({\hat{\uptheta}}_{\textrm{S}}\right),{\Sigma}_{\textrm{W}}\left({\hat{\uptheta}}_{\textrm{S}}\right)\right]$$

(A11)

$${df}_{S\_B,I\_W}={df}_{W,\, Saturated}-{df}_{W,\, independent}$$

(A12)

TLI

The TLI is classified as a non-normed fit index that penalizes for lack of parsimony of the model (Tucker & Lewis, 1973). The TLI_{PS_B} evaluates the goodness-of-fit of the between-level model by comparing the hypothesized between-level model and the independent between-level model when the within-level model is saturated. In contrast, the TLI_{PS_W} estimates the goodness-of-fit of the within-level model by comparing the hypothesized within-level model and the independent within-level model when the between-level model is saturated. The formulas for TLI_{PS _ B} and TLI_{PS _ W} could be derived in terms of A13 and A14 respectively, as follows:

$${TLI}_{PS\_B}=\frac{\frac{{\chi^2}_{I\_B,\;S\_W}}{df_{I\_B,\;S\_W}}-\frac{\chi_{PS\_B}^2}{df_{PS\_B}}}{\frac{{\chi^2}_{I\_B,\;S\_W}}{df_{I\_B,\;S\_W}}-1}$$

(A13)

$${TLI}_{PS\_W}=\frac{\frac{{\chi^2}_{S\_B,\;I\_W}}{df_{S\_B,\;I\_W}}-\frac{\chi_{PS\_W}^2}{df_{PS\_W}}}{\frac{{\chi^2}_{S\_B,\;I\_W}}{df_{S\_B,\;I\_W}}-1}$$

(A14)

SRMR

SRMR can be classified into the SRMR_W at the within-level and the SRMR_B at the between-level models, respectively. Note that SRMR is independent of χ² statistics and can be defined as the deviation between the sample variance-covariance matrix and the reproduced variance-covariance matrix. Statistical packages, such as Mplus (Muthen & Muthen, 2017), can provide SRMR_W and SRMR_B for model evaluations. Moreover, SRMR_B can be regarded as the normed average distance between the sample variance matrix of observed variables and model-implied variance matrix at the between level. Therefore, the SRMR_B can be derived in terms of A15 as follows:

$${SRMR}_B=\sqrt{\frac{2\sum_{\textrm{i}=1}^{\mathrm{p}}\sum_{\textrm{j}=1}^{\mathrm{i}}{\left[\frac{\left({\Sigma}_{{\textrm{B}}_{\mathrm{i}\textrm{j}}}-{\Sigma}_{\textrm{B}}\ {\left(\uptheta \right)}_{\textrm{i}\mathrm{j}}\right)}{\Sigma_{{\textrm{B}}_{\mathrm{i}\textrm{i}}}{\Sigma}_{{\textrm{B}}_{\mathrm{j}\textrm{j}}}}\right]}^2}{\textrm{p}\ \left(\textrm{p}+1\right)}}$$

(A15)

Using the similar definition, SRMR_W can be defined as the normed average distance between the sample variance matrix and model-implied variance matrix at the within level. Therefore, the SRMR_W can be derived in terms of A16 as follows:

$${SRMR}_W=\sqrt{\frac{2\sum_{\textrm{i}=1}^{\textrm{p}}\sum_{\textrm{j}=1}^{\mathrm{i}}{\left[\frac{\left({\Sigma}_{{\textrm{W}}_{\mathrm{i}\textrm{j}}}-{\Sigma}_{\textrm{W}}\ {\left(\uptheta \right)}_{\textrm{i}\mathrm{j}}\right)}{\Sigma_{{\textrm{W}}_{\mathrm{i}\textrm{i}}}{\Sigma}_{{\textrm{W}}_{\mathrm{j}\textrm{j}}}}\right]}^2}{\textrm{p}\ \left(\textrm{p}+1\right)}}$$

(A16)

Appendix B. Results of the additional simulations

Table 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 and 21.

Table 4 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC1 (.091) for THR = 50:50 and FL= .40

Table 5 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC1 (.091) for THR = 50:50 and FL= .55

Table 6 Means of level-specific χ² test statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC1 (.091) for THR = 50:50 and FL= .70

Table 7 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC1 (.091) for THR = 80:20 and FL= .40

Table 8 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC1 (.091) for THR = 80:20 and FL= .55

Table 9 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC1 (.091) for THR = 80:20 and FL= .70

Table 10 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC2 (.231) for THR = 50:50 and FL= .40

Table 11 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC2 (.231) for THR = 50:50 and FL= .55

Table 12 Means of level-specific χ² test statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC2 (.231) for THR = 50:50 and FL= .70

Table 13 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC2 (.231) for THR = 80:20 and FL= .40

Table 14 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC2 (.231) for THR = 80:20 and FL= .55

Table 15 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC2 (.231) for THR = 80:20 and FL= .70

Table 16 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC3 (.333) for THR = 50:50 and FL= .40

Table 17 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC3 (.333) for THR = 50:50 and FL= .55

Table 18 Means of level-specific χ² test statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC3 (.333) for THR = 50:50 and FL= .70.

Table 19 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC3 (.333) for THR = 80:20 and FL= .40

Table 20 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC3 (.333) for THR = 80:20 and FL= .55

Table 21 Means of level-specific χ² statistics and fit indices for the M_C, M_B, and M_W within a condition of ICC3 (.333) for THR = 80:20 and FL= .70