In measurement invariance testing, when a certain level of full invariance is not achieved, the sequential backward specification search method with the largest modification index (SBSS_LMFI) is often used to identify the source of non-invariance. SBSS_LMFI has been studied under complete data but not missing data. Focusing on Likert-type scale variables, this study examined two methods for dealing with missing data in SBSS_LMFI using Monte Carlo simulation: robust full information maximum likelihood estimator (rFIML) and mean and variance adjusted weighted least squared estimator coupled with pairwise deletion (WLSMV_PD). The result suggests that WLSMV_PD could result in not only over-rejections of invariance models but also reductions of power to identify non-invariant items. In contrast, rFIML provided good control of type I error rates, although it required a larger sample size to yield sufficient power to identify non-invariant items. Recommendations based on the result were provided.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. Advanced structural equation modeling: Issues and techniques,243, 277.
Asparouhov, T., & Muthén, B. O. (2010). Multiple imputation with Mplus.MPlus Web Notes. Retrieved from https://www.statmodel.com/download/Imputations7.pdf
Beam, C. R., Marcus, K., Turkheimer, E., & Emery, R. E. (2018). Gender differences in the structure of marital quality. Behavior genetics, 48(3), 209–223. https://doi.org/10.1007/s10519-018-9892-4
Belzak, W., & Bauer, D. J. (2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods. https://doi.org/10.1037/met0000253
Bollen, K.A. (1989). Structural equations with latent variables. New York, NY: Wiley
Bou Malham, P., & Saucier, G. (2014). Measurement invariance of social axioms in 23 countries. Journal of Cross-Cultural Psychology, 45(7), 1046–1060. https://doi.org/10.1177/0022022114534771
Brown, T. A. (2014). Confirmatory factor analysis for applied research. New York, NY: Guilford Publications.
Byrne, B. M. (2013). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York, NY: Routledge.
Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: the issue of partial measurement invariance. Psychological Bulletin, 105(3), 456. https://doi.org/10.1037/0033-2909.105.3.456
Chan, M. H. M., Gerhardt, M., & Feng, X. (2019). Measurement invariance across age groups and over 20 years’ time of the Negative and Positive Affect Scale (NAPAS). European Journal of Psychological Assessment. https://doi.org/10.1027/1015-5759/a000529
Chen, F. F. (2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of personality and social psychology, 95(5), 1005. https://doi.org/10.1037/e514412014-064
Chen, P. Y., Wu, W., Garnier-Villarreal, M., Kite, B. A., & Jia, F. (2019). Testing measurement invariance with ordinal missing data: A comparison of estimators and missing data techniques. Multivariate behavioral research, 1–15. https://doi.org/10.1080/00273171.2019.1608799
Chou, C. P., & Bentler, P. M. (2002). Model modification in structural equation modeling by imposing constraints. Computational statistics & data analysis, 41(2), 271–287
DiStefano, C., & Morgan, G. B. (2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling, 21(3), 425–438. https://doi.org/10.1080/10705511.2014.915373
Erreygers, S., Vandebosch, H., Vranjes, I., Baillien, E., & De Witte, H. (2018). Development of a measure of adolescents’ online prosocial behavior. Journal of Children and Media, 12(4), 448–464. https://doi.org/10.1080/17482798.2018.1431558
Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457. https://doi.org/10.1207/s15328007sem0803_5
Enders, C. K., & Gottschall, A. C. (2011). Multiple imputation strategies for multiple group structural equation models. Structural Equation Modeling, 18(1), 35–54.
Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466. https://doi.org/10.1037/1082-989x.9.4.466
Fokkema, M., Smits, N., Kelderman, H., & Cuijpers, P. (2013). Response shifts in mental health interventions: An illustration of longitudinal measurement invariance. Psychological Assessment, 25(2), 520. https://doi.org/10.1037/a0031669
French, B. F., & Finch, H. (2016). Factorial invariance testing under different levels of partial loading invariance within a multiple group confirmatory factor analysis model. Journal of Modern Applied Statistical Methods, 15(1), 26. https://doi.org/10.22237/jmasm/1462076700.
Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical care, 44(11 Suppl 3), S78. https://doi.org/10.1097/01.mlr.0000245454.12228.8f
Graham, J. W. (2003). Adding missing-data-relevant variables to FIML-based structural equation models. Structural Equation Modeling, 10(1), 80–100. https://doi.org/10.1207/s15328007sem1001_4
Hakkarainen, A. M., Holopainen, L. K., & Savolainen, H. K. (2016). The impact of learning difficulties and socioemotional and behavioural problems on transition to postsecondary education or work life in Finland: a five-year follow-up study. European Journal of Special Needs Education, 31(2), 171–186
Huang, P. H. (2018). A penalized likelihood method for multi‐group structural equation modelling. British Journal of Mathematical and Statistical Psychology, 71(3), 499–522. https://doi.org/10.1111/bmsp.12130
Huang, P. H. (2020) lslx: Semi-confirmatory structural equation modeling via penalized likelihood. Journal of Statistical Software. https://doi.org/10.18637/jss.v093.i07
Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized structural equation modeling. Structural equation modeling: a multidisciplinary journal, 23(4), 555–566. https://doi.org/10.1080/10705511.2016.1154793
Johnson, E. C., Meade, A. W., & DuVernet, A. M. (2009). The role of referent indicators in tests of measurement invariance. Structural Equation Modeling, 16(4), 642–657. https://doi.org/10.1080/10705510903206014
Jung, E., & Yoon, M. (2016). Comparisons of three empirical methods for partial factorial invariance: forward, backward, and factor-ratio tests. Structural Equation Modeling: A Multidisciplinary Journal, 23(4), 567–584. https://doi.org/10.1080/10705511.2015.1138092
Jung, E., & Yoon, M. (2017). Two-step approach to partial factorial invariance: Selecting a reference variable and identifying the source of noninvariance. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 65–79. https://doi.org/10.1080/10705511.2016.1251845
Kim, G., Wang, S. Y., & Sellbom, M. (2018). Measurement Equivalence of the Subjective Well-Being Scale Among Racially/Ethnically Diverse Older Adults. The Journals of Gerontology: Series B. https://doi.org/10.1093/geronb/gby110
Li, C. H. (2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093
Liang, X., & Jacobucci, R. (2019). Regularized Structural Equation Modeling to Detect Measurement Bias: Evaluation of Lasso, Adaptive Lasso, and Elastic Net. Structural Equation Modeling: A Multidisciplinary Journal. https://doi.org/10.1080/10705511.2019.1693273
Liu, Y., Millsap, R. E., West, S. G., Tein, J. Y., Tanaka, R., & Grimm, K. J. (2017). Testing Measurement Invariance in Longitudinal Data With Ordered-Categorical Measures. Psychological Methods. 22(3), 486–506. https://doi.org/10.1037/met0000075
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: the problem of capitalization on chance. Psychological Bulletin, 111(3), 490. https://doi.org/10.1037//0033-2909.111.3.490
McNeish, D. (2016). On using Bayesian methods to address small sample problems. Structural Equation Modeling: A Multidisciplinary Journal, 23(5), 750–773. https://doi.org/10.1080/10705511.2016.1186549
Meade, A. W., & Bauer, D. J. (2007). Power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 611–635. https://doi.org/10.1080/10705510701575461
Meade, A. W., & Lautenschlager, G. J. (2004). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11(1), 60–72. https://doi.org/10.1207/s15328007sem1101_5
Millsap, R. E. (2012). Statistical approaches to measurement invariance. New York, NY: Routledge.
Millsap, R. E., & Kwok, O. M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological methods, 9(1), 93. https://doi.org/10.1037/1082-989x.9.1.93
Muthén, B. O. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132. https://doi.org/10.1007/bf02294210
Muthén, B. O. (2017, Feb 26). Multiple imputation [Online forum comment]. Message posted www.statmodel.com/discussion/ messages/22 /381.html?1488130113
Muthén, L.K (2011, Jul, 26) Modification indices. [Online forum comment]. Message posted http://www.statmodel.com/discussion/messages/9/153.html?1457176945
Muthén, B., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus web notes, 4(5), 1–22. Retrieved from https://www.statmodel.com/download/webnotes/CatMGLong.pdf
Muthén, B.O., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes Unpublished Technical Report. Retrieved from https://www.statmodel.com/download/Article_075.pdf
Muthén, L. K. & Muthén, B. O. (1998–2017). Mplus user’s guide. Eighth Edition. Los Angeles, CA: Author.
Muthén, B.O, Muthén, L.K & Asparouhov, T. (2015). Estimator choices with categorical outcomes. Mplus Web Notes: March 2015. Retrieved from https://www.statmodel.com/download/EstimatorChoices.pdf
Nye, C. D., Bradburn, J., Olenick, J., Bialko, C., & Drasgow, F. (2019). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22(3), 678–709. https://doi.org/10.1177/1094428118761122
Nye, C. D., & Drasgow, F. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96(5), 966. https://doi.org/10.1037/a0022955
Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44(4), 443–460. https://doi.org/10.1007/bf02296207
Oort, F. J. (1998). Simulation study of item bias detection with restricted factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 5(2), 107–124. https://doi.org/10.1080/10705519809540095
Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315
Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling, 21(2), 167–180. https://doi.org/10.1080/10705511.2014.882658
Savalei, V. (2014). Understanding robust corrections in structural equation modeling. Structural Equation Modeling, 21(1), 149–160. https://doi.org/10.1080/10705511.2013.824793
Savalei, V., & Bentler, P. M. (2005). A statistically justified pairwise ML method for incomplete nonnormal data: A comparison with direct ML and pairwise ADF. Structural Equation Modeling, 12(2), 183–214. https://doi.org/10.1207/s15328007sem1202_1
Savalei, V., & Falk, C. F. (2014). Robust two-stage approach outperforms robust full information maximum likelihood with incomplete nonnormal data. Structural Equation Modeling, 21(2), 280–302. https://doi.org/10.1080/10705511.2014.882692
Sörbom, D. (1989). Model modification. Psychometrika, 54(3), 371–384. https://doi.org/10.1007/BF02294623
Sommer, K., et al.. (2019). Consistency matters: measurement invariance of the EORTC QLQ-C30 questionnaire in patients with hematologic malignancies. Quality of Life Research, 1–9. https://doi.org/10.1007/s11136-019-02369-5
Schmitt, N., & Kuljanin, G. (2008). Measurement invariance: Review of practice and implications. Human Resource Management Review, 18(4), 210–222. https://doi.org/10.1016/j.hrmr.2008.03.003
Shi, D., Song, H., & Lewis, M. D. (2017a). The impact of partial factorial invariance on cross-group comparisons. Assessment, https://doi.org/10.1177/1073191117711020
Shi, D., Song, H., Liao, X., Terry, R., & Snyder, L. A. (2017b). Bayesian SEM for specification search problems in testing factorial invariance. Multivariate Behavioral Research, 52(4), 430–444. https://doi.org/10.1080/00273171.2017.1306432
Steinmetz, H. (2013). Analyzing observed composite differences across groups. Methodology, 9, 1–12. https://doi.org/10.1027/1614-2241/a000049
Suh, Y. (2015). The performance of maximum likelihood and weighted least square mean and variance adjusted estimators in testing differential item functioning with nonnormal trait distributions. Structural Equation Modeling, 22(4), 568–580. https://doi.org/10.1080/10705511.2014.937669
Teman, E.D. (2012). The performance of multiple imputation and full information maximum likelihood for missing ordinal data in structural equation models. Ann Arbor, MI: ProQuest.
Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
Willoughby, M. T., Pek, J., Greenberg, M. T., & Family Life Project Investigators. (2012). Parent-reported attention deficit/hyperactivity symptomatology in preschool-aged children: Factor structure, developmental change, and early risk factors. Journal of abnormal child psychology, 40(8), 1301–1312. https://doi.org/10.1007/s10802-012-9641-8
Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: current approaches and future directions. Psychological Methods, 12(1), 58. https://doi.org/10.1037/1082-989x.12.1.58
Whittaker, T. A., & Khojasteh, J. (2013). A comparison of methods to detect invariant reference indicators in structural equation modelling. International Journal of Quantitative Research in Education, 1(4), 426–443. https://doi.org/10.1504/ijqre.2013.058310
Wu, W., Jia, F., & Enders, C. (2015). A comparison of imputation strategies for ordinal missing data on Likert scale variables. Multivariate Behavioral Research, 50(5), 484–503. https://doi.org/10.1080/00273171.2015.1022644
Xu, Y, Green, S.B. (2016). The impact of varying the number of measurement invariance constrains on the assessment of between group differences of latent means. Structural equation modeling, 23(2), 290–301. https://doi.org/10.1080/10705511.2015.1047932
Yuan, K. H., & Bentler, P. M. (2000). Three likelihood‐based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30(1), 165–200. https://doi.org/10.1111/0081-1750.00078
Yoon, M., & Kim, E. S. (2014). A comparison of sequential and nonsequential specification searches in testing factorial invariance. Behavior research methods, 46(4), 1199–1206. https://doi.org/10.3758/s13428-013-0430-2
Yoon, M., & Millsap, R. E. (2007). Detecting violations of factorial invariance using data-based specification searches: A Monte Carlo study. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 435–463. https://doi.org/10.1080/10705510701301677
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
About this article
Cite this article
Chen, PY., Wu, W., Brandt, H. et al. Addressing missing data in specification search in measurement invariance testing with Likert-type scale variables: A comparison of two approaches. Behav Res 52, 2567–2587 (2020). https://doi.org/10.3758/s13428-020-01415-2
- Specification search
- Partial invariance model
- Ordinal missing data
- Measurement invaraince
- Modification index