Addressing missing data in specification search in measurement invariance testing with Likert-type scale variables: A comparison of two approaches

Abstract

In measurement invariance testing, when a certain level of full invariance is not achieved, the sequential backward specification search method with the largest modification index (SBSS_LMFI) is often used to identify the source of non-invariance. SBSS_LMFI has been studied under complete data but not missing data. Focusing on Likert-type scale variables, this study examined two methods for dealing with missing data in SBSS_LMFI using Monte Carlo simulation: robust full information maximum likelihood estimator (rFIML) and mean and variance adjusted weighted least squared estimator coupled with pairwise deletion (WLSMV_PD). The result suggests that WLSMV_PD could result in not only over-rejections of invariance models but also reductions of power to identify non-invariant items. In contrast, rFIML provided good control of type I error rates, although it required a larger sample size to yield sufficient power to identify non-invariant items. Recommendations based on the result were provided.

This is a preview of subscription content, access via your institution.

Fig. 1.
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. Advanced structural equation modeling: Issues and techniques,243, 277.

    Google Scholar 

  2. Asparouhov, T., & Muthén, B. O. (2010). Multiple imputation with Mplus.MPlus Web Notes. Retrieved from https://www.statmodel.com/download/Imputations7.pdf

  3. Beam, C. R., Marcus, K., Turkheimer, E., & Emery, R. E. (2018). Gender differences in the structure of marital quality. Behavior genetics, 48(3), 209–223. https://doi.org/10.1007/s10519-018-9892-4

    Article  PubMed  PubMed Central  Google Scholar 

  4. Belzak, W., & Bauer, D. J. (2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods. https://doi.org/10.1037/met0000253

  5. Bollen, K.A. (1989). Structural equations with latent variables. New York, NY: Wiley

    Google Scholar 

  6. Bou Malham, P., & Saucier, G. (2014). Measurement invariance of social axioms in 23 countries. Journal of Cross-Cultural Psychology, 45(7), 1046–1060. https://doi.org/10.1177/0022022114534771

    Article  Google Scholar 

  7. Brown, T. A. (2014). Confirmatory factor analysis for applied research. New York, NY: Guilford Publications.

    Google Scholar 

  8. Byrne, B. M. (2013). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York, NY: Routledge.

    Google Scholar 

  9. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: the issue of partial measurement invariance. Psychological Bulletin, 105(3), 456. https://doi.org/10.1037/0033-2909.105.3.456

    Article  Google Scholar 

  10. Chan, M. H. M., Gerhardt, M., & Feng, X. (2019). Measurement invariance across age groups and over 20 years’ time of the Negative and Positive Affect Scale (NAPAS). European Journal of Psychological Assessment. https://doi.org/10.1027/1015-5759/a000529

  11. Chen, F. F. (2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of personality and social psychology, 95(5), 1005. https://doi.org/10.1037/e514412014-064

    Article  PubMed  Google Scholar 

  12. Chen, P. Y., Wu, W., Garnier-Villarreal, M., Kite, B. A., & Jia, F. (2019). Testing measurement invariance with ordinal missing data: A comparison of estimators and missing data techniques. Multivariate behavioral research, 1–15. https://doi.org/10.1080/00273171.2019.1608799

  13. Chou, C. P., & Bentler, P. M. (2002). Model modification in structural equation modeling by imposing constraints. Computational statistics & data analysis, 41(2), 271–287

    Article  Google Scholar 

  14. DiStefano, C., & Morgan, G. B. (2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling, 21(3), 425–438. https://doi.org/10.1080/10705511.2014.915373

    Article  Google Scholar 

  15. Erreygers, S., Vandebosch, H., Vranjes, I., Baillien, E., & De Witte, H. (2018). Development of a measure of adolescents’ online prosocial behavior. Journal of Children and Media, 12(4), 448–464. https://doi.org/10.1080/17482798.2018.1431558

    Article  Google Scholar 

  16. Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457. https://doi.org/10.1207/s15328007sem0803_5

    Article  Google Scholar 

  17. Enders, C. K., & Gottschall, A. C. (2011). Multiple imputation strategies for multiple group structural equation models. Structural Equation Modeling, 18(1), 35–54.

    Article  Google Scholar 

  18. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466. https://doi.org/10.1037/1082-989x.9.4.466

    Article  PubMed  PubMed Central  Google Scholar 

  19. Fokkema, M., Smits, N., Kelderman, H., & Cuijpers, P. (2013). Response shifts in mental health interventions: An illustration of longitudinal measurement invariance. Psychological Assessment, 25(2), 520. https://doi.org/10.1037/a0031669

    Article  PubMed  Google Scholar 

  20. French, B. F., & Finch, H. (2016). Factorial invariance testing under different levels of partial loading invariance within a multiple group confirmatory factor analysis model. Journal of Modern Applied Statistical Methods, 15(1), 26. https://doi.org/10.22237/jmasm/1462076700.

    Article  Google Scholar 

  21. Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical care, 44(11 Suppl 3), S78. https://doi.org/10.1097/01.mlr.0000245454.12228.8f

    Article  PubMed  PubMed Central  Google Scholar 

  22. Graham, J. W. (2003). Adding missing-data-relevant variables to FIML-based structural equation models. Structural Equation Modeling, 10(1), 80–100. https://doi.org/10.1207/s15328007sem1001_4

    Article  Google Scholar 

  23. Hakkarainen, A. M., Holopainen, L. K., & Savolainen, H. K. (2016). The impact of learning difficulties and socioemotional and behavioural problems on transition to postsecondary education or work life in Finland: a five-year follow-up study. European Journal of Special Needs Education, 31(2), 171–186

    Article  Google Scholar 

  24. Huang, P. H. (2018). A penalized likelihood method for multi‐group structural equation modelling. British Journal of Mathematical and Statistical Psychology, 71(3), 499–522. https://doi.org/10.1111/bmsp.12130

    Article  PubMed  Google Scholar 

  25. Huang, P. H. (2020) lslx: Semi-confirmatory structural equation modeling via penalized likelihood. Journal of Statistical Software. https://doi.org/10.18637/jss.v093.i07

  26. Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized structural equation modeling. Structural equation modeling: a multidisciplinary journal, 23(4), 555–566. https://doi.org/10.1080/10705511.2016.1154793

    Article  Google Scholar 

  27. Johnson, E. C., Meade, A. W., & DuVernet, A. M. (2009). The role of referent indicators in tests of measurement invariance. Structural Equation Modeling, 16(4), 642–657. https://doi.org/10.1080/10705510903206014

    Article  Google Scholar 

  28. Jung, E., & Yoon, M. (2016). Comparisons of three empirical methods for partial factorial invariance: forward, backward, and factor-ratio tests. Structural Equation Modeling: A Multidisciplinary Journal, 23(4), 567–584. https://doi.org/10.1080/10705511.2015.1138092

    Article  Google Scholar 

  29. Jung, E., & Yoon, M. (2017). Two-step approach to partial factorial invariance: Selecting a reference variable and identifying the source of noninvariance. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 65–79. https://doi.org/10.1080/10705511.2016.1251845

    Article  Google Scholar 

  30. Kim, G., Wang, S. Y., & Sellbom, M. (2018). Measurement Equivalence of the Subjective Well-Being Scale Among Racially/Ethnically Diverse Older Adults. The Journals of Gerontology: Series B. https://doi.org/10.1093/geronb/gby110

  31. Li, C. H. (2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093

    Article  PubMed  Google Scholar 

  32. Liang, X., & Jacobucci, R. (2019). Regularized Structural Equation Modeling to Detect Measurement Bias: Evaluation of Lasso, Adaptive Lasso, and Elastic Net. Structural Equation Modeling: A Multidisciplinary Journal. https://doi.org/10.1080/10705511.2019.1693273

  33. Liu, Y., Millsap, R. E., West, S. G., Tein, J. Y., Tanaka, R., & Grimm, K. J. (2017). Testing Measurement Invariance in Longitudinal Data With Ordered-Categorical Measures. Psychological Methods. 22(3), 486–506. https://doi.org/10.1037/met0000075

    Article  PubMed  Google Scholar 

  34. MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: the problem of capitalization on chance. Psychological Bulletin, 111(3), 490. https://doi.org/10.1037//0033-2909.111.3.490

    Article  PubMed  Google Scholar 

  35. McNeish, D. (2016). On using Bayesian methods to address small sample problems. Structural Equation Modeling: A Multidisciplinary Journal, 23(5), 750–773. https://doi.org/10.1080/10705511.2016.1186549

    Article  Google Scholar 

  36. Meade, A. W., & Bauer, D. J. (2007). Power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 611–635. https://doi.org/10.1080/10705510701575461

    Article  Google Scholar 

  37. Meade, A. W., & Lautenschlager, G. J. (2004). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11(1), 60–72. https://doi.org/10.1207/s15328007sem1101_5

    Article  Google Scholar 

  38. Millsap, R. E. (2012). Statistical approaches to measurement invariance. New York, NY: Routledge.

    Google Scholar 

  39. Millsap, R. E., & Kwok, O. M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological methods, 9(1), 93. https://doi.org/10.1037/1082-989x.9.1.93

    Article  PubMed  Google Scholar 

  40. Muthén, B. O. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132. https://doi.org/10.1007/bf02294210

    Article  Google Scholar 

  41. Muthén, B. O. (2017, Feb 26). Multiple imputation [Online forum comment]. Message posted www.statmodel.com/discussion/ messages/22 /381.html?1488130113

  42. Muthén, L.K (2011, Jul, 26) Modification indices. [Online forum comment]. Message posted http://www.statmodel.com/discussion/messages/9/153.html?1457176945

  43. Muthén, B., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus web notes, 4(5), 1–22. Retrieved from https://www.statmodel.com/download/webnotes/CatMGLong.pdf

    Google Scholar 

  44. Muthén, B.O., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes Unpublished Technical Report. Retrieved from https://www.statmodel.com/download/Article_075.pdf

  45. Muthén, L. K. & Muthén, B. O. (1998–2017). Mplus user’s guide. Eighth Edition. Los Angeles, CA: Author.

  46. Muthén, B.O, Muthén, L.K & Asparouhov, T. (2015). Estimator choices with categorical outcomes. Mplus Web Notes: March 2015. Retrieved from https://www.statmodel.com/download/EstimatorChoices.pdf

  47. Nye, C. D., Bradburn, J., Olenick, J., Bialko, C., & Drasgow, F. (2019). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22(3), 678–709. https://doi.org/10.1177/1094428118761122

    Article  Google Scholar 

  48. Nye, C. D., & Drasgow, F. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96(5), 966. https://doi.org/10.1037/a0022955

    Article  PubMed  Google Scholar 

  49. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44(4), 443–460. https://doi.org/10.1007/bf02296207

    Article  Google Scholar 

  50. Oort, F. J. (1998). Simulation study of item bias detection with restricted factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 5(2), 107–124. https://doi.org/10.1080/10705519809540095

    Article  Google Scholar 

  51. Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004

    Article  PubMed  Google Scholar 

  52. R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/

  53. Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315

    Article  PubMed  Google Scholar 

  54. Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling, 21(2), 167–180. https://doi.org/10.1080/10705511.2014.882658

    Article  Google Scholar 

  55. Savalei, V. (2014). Understanding robust corrections in structural equation modeling. Structural Equation Modeling, 21(1), 149–160. https://doi.org/10.1080/10705511.2013.824793

    Article  Google Scholar 

  56. Savalei, V., & Bentler, P. M. (2005). A statistically justified pairwise ML method for incomplete nonnormal data: A comparison with direct ML and pairwise ADF. Structural Equation Modeling, 12(2), 183–214. https://doi.org/10.1207/s15328007sem1202_1

    Article  Google Scholar 

  57. Savalei, V., & Falk, C. F. (2014). Robust two-stage approach outperforms robust full information maximum likelihood with incomplete nonnormal data. Structural Equation Modeling, 21(2), 280–302. https://doi.org/10.1080/10705511.2014.882692

  58. Sörbom, D. (1989). Model modification. Psychometrika, 54(3), 371–384. https://doi.org/10.1007/BF02294623

    Article  Google Scholar 

  59. Sommer, K., et al.. (2019). Consistency matters: measurement invariance of the EORTC QLQ-C30 questionnaire in patients with hematologic malignancies. Quality of Life Research, 1–9. https://doi.org/10.1007/s11136-019-02369-5

  60. Schmitt, N., & Kuljanin, G. (2008). Measurement invariance: Review of practice and implications. Human Resource Management Review, 18(4), 210–222. https://doi.org/10.1016/j.hrmr.2008.03.003

    Article  Google Scholar 

  61. Shi, D., Song, H., & Lewis, M. D. (2017a). The impact of partial factorial invariance on cross-group comparisons. Assessment, https://doi.org/10.1177/1073191117711020

  62. Shi, D., Song, H., Liao, X., Terry, R., & Snyder, L. A. (2017b). Bayesian SEM for specification search problems in testing factorial invariance. Multivariate Behavioral Research, 52(4), 430–444. https://doi.org/10.1080/00273171.2017.1306432

    Article  PubMed  Google Scholar 

  63. Steinmetz, H. (2013). Analyzing observed composite differences across groups. Methodology, 9, 1–12. https://doi.org/10.1027/1614-2241/a000049

  64. Suh, Y. (2015). The performance of maximum likelihood and weighted least square mean and variance adjusted estimators in testing differential item functioning with nonnormal trait distributions. Structural Equation Modeling, 22(4), 568–580. https://doi.org/10.1080/10705511.2014.937669

  65. Teman, E.D. (2012). The performance of multiple imputation and full information maximum likelihood for missing ordinal data in structural equation models. Ann Arbor, MI: ProQuest.

    Google Scholar 

  66. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002

    Article  Google Scholar 

  67. Willoughby, M. T., Pek, J., Greenberg, M. T., & Family Life Project Investigators. (2012). Parent-reported attention deficit/hyperactivity symptomatology in preschool-aged children: Factor structure, developmental change, and early risk factors. Journal of abnormal child psychology, 40(8), 1301–1312. https://doi.org/10.1007/s10802-012-9641-8

    Article  PubMed  PubMed Central  Google Scholar 

  68. Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: current approaches and future directions. Psychological Methods, 12(1), 58. https://doi.org/10.1037/1082-989x.12.1.58

    Article  PubMed  PubMed Central  Google Scholar 

  69. Whittaker, T. A., & Khojasteh, J. (2013). A comparison of methods to detect invariant reference indicators in structural equation modelling. International Journal of Quantitative Research in Education, 1(4), 426–443. https://doi.org/10.1504/ijqre.2013.058310

    Article  Google Scholar 

  70. Wu, W., Jia, F., & Enders, C. (2015). A comparison of imputation strategies for ordinal missing data on Likert scale variables. Multivariate Behavioral Research, 50(5), 484–503. https://doi.org/10.1080/00273171.2015.1022644

    Article  PubMed  Google Scholar 

  71. Xu, Y, Green, S.B. (2016). The impact of varying the number of measurement invariance constrains on the assessment of between group differences of latent means. Structural equation modeling, 23(2), 290–301. https://doi.org/10.1080/10705511.2015.1047932

    Article  Google Scholar 

  72. Yuan, K. H., & Bentler, P. M. (2000). Three likelihood‐based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30(1), 165–200. https://doi.org/10.1111/0081-1750.00078

    Article  Google Scholar 

  73. Yoon, M., & Kim, E. S. (2014). A comparison of sequential and nonsequential specification searches in testing factorial invariance. Behavior research methods, 46(4), 1199–1206. https://doi.org/10.3758/s13428-013-0430-2

    Article  PubMed  Google Scholar 

  74. Yoon, M., & Millsap, R. E. (2007). Detecting violations of factorial invariance using data-based specification searches: A Monte Carlo study. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 435–463. https://doi.org/10.1080/10705510701301677

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Po-Yi Chen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOCX 1544 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chen, PY., Wu, W., Brandt, H. et al. Addressing missing data in specification search in measurement invariance testing with Likert-type scale variables: A comparison of two approaches. Behav Res 52, 2567–2587 (2020). https://doi.org/10.3758/s13428-020-01415-2

Download citation

Keywords

  • Specification search
  • Partial invariance model
  • Ordinal missing data
  • Measurement invaraince
  • Modification index