Skip to main content
SpringerLink
Log in

Understanding non-linear modeling of measurement invariance in heterogeneous populations

  • Regular Article
  • Published:
Advances in Data Analysis and Classification Aims and scope Submit manuscript

Abstract

This study examined how a non-linear modeling of ordered categorical variables within multiple-group confirmatory factor analysis supported measurement invariance. A four-item classroom disciplinary climate scale used in cross-cultural framework was empirically investigated. In the first part of the analysis, a separated categorical confirmatory factor analysis was initially applied to account for the complex structure of the relationships between the observed measures in each country. The categorical multiple-group confirmatory factor analysis (MGCFA) was then used to conduct a cross-country examination of full measurement invariance namely the configural, metric, and scalar levels of invariance in the classroom discipline climate measures. The categorical MGCFA modeling supported configural and metric invariances as well as scalar invariance for the latent factor structure of classroom disciplinary climate. This finding implying meaningful cross-country comparisons on the scale means, on the associations of classroom disciplinary climate scale with other scales and on the item-factor latent structure. Application of the categorical modeling appeared to correctly specify the factor structure of the scale, thereby promising the appropriateness of reporting comparisons such as rankings of many groups, and illustrating league tables of different heterogeneous groups. Limitations of the modeling in this study and future suggestions for measurement invariance testing in studies with large numbers of groups are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. A categorical variable is one that has two or more categories (or values), but there is no intrinsic ordering to the categories. Gender is an example. A continuous variable is one that can, within a given range, take on numerous values. Age is an example.

  2. Gonzalez (2012) provides a detailed explanation of the mechanics of rescaling weights using SPSS macros.

  3. The unequal weight effect—the effect of unequal weight variation on the precision of the estimates of the weights (Biemer and Christ 2008)—is approximately computed at 1.72, meaning that variability of the computed weights that could lead to poor estimates is insubstantial.

  4. Factor models are subscripted with a g to indicate that parameters may take different values across groups, thus permitting comparison of group differences.

  5. Group 22 has smaller sample size compared to the other groups, and was observed with the highest chi-square value. The RMSEA was high and its wider confidence interval is affected by the size of the sample above and beyond the number of parameters.

  6. Bayesian structural equation modeling measurement invariance and multiple group factor analysis alignment are available in Mplus 7.1 for continuous and binary variables and their extensions to ordered-categorical variables from Mplus 7.3.

References

  • Anderson TW, Rubin H (1956) Statistical inference in factor analysis. In: Neyman J (ed) Third Berkeley symposium on mathematical statistics and probability, vol 3. University of California Press, Berkeley, pp 111–150

    Google Scholar 

  • Asparouhov T (2005) Sampling weights in latent variable modeling. Struct Equ Model Multidiscip J 12(3):411–434

    Article  MathSciNet  Google Scholar 

  • Asparouhov T, Muthén B (2013) Multiple Group Factor Analysis Alignment. Mplus Web Notes

  • Asparouhov T, Muthén B (2014) Multiple-group factor analysis alignment. Struct Equ Model: Multidiscip J 21:1–14

    Article  MathSciNet  Google Scholar 

  • Babakus E, Ferguson CE, Jöreskog KG (1987) The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. J Mark Res 24(2):222–228

    Article  Google Scholar 

  • Bauer DJ (2005) The role of nonlinear factor-to-indicator relationships in tests of measurement equivalence. Psychol Methods 10:305–316

    Article  Google Scholar 

  • Bentler P, Satorra A (2010) Testing model nesting and equivalence. Psychol Methods 15(2):111–123

    Article  Google Scholar 

  • Biemer PP, Christ SL (2008) Weighting survey data. In: Hox J, Dillman D (eds) International handbook on survey methodology. Lawrence Erlbaum Associates, Mahwah, pp 317–341

    Google Scholar 

  • Brown TA (2006) Confirmatory factor analysis. The Guilford Press, New York

    Google Scholar 

  • Browne MW, Arminger G (1995) Specification and estimation of mean- and covariance-structure models. In: Arminger G, Clogg CC, Sobel ME (eds) Handbook of statistical modeling for the social and behavioral sciences. Plenum Press, New York, pp 311–359

    Chapter  Google Scholar 

  • Browne MW, Cudeck R (1992) Alternative ways of assessing model fit. Sociol Methods Res 21:230–258

    Article  Google Scholar 

  • Bryant FB, Satorra A (2012) Principles and practice of scaled difference chi-square testing. Struct Equ Model Multidiscip J 19:372–398

    Article  MathSciNet  Google Scholar 

  • Byrne BM (2008) Testing for multigroup equivalence of a measuring instrument: a walk through the process. Psicothema 20:872–882

    Google Scholar 

  • Chen FF (2007) Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Model Multidiscip J 14(3):464–504

    Article  MathSciNet  Google Scholar 

  • Cheung GW, Rensvold RB (2002) Evaluating goodness-of-fit indexes for testing measurement invariance. Struct Equ Model Multidiscip J 9(2):233–255

    Article  MathSciNet  Google Scholar 

  • Cieciuch J, Davidov E, Schmidt P, Algesheimer R, Schwartz SH (2014) Comparing results of an exact vs. an approximate (Bayesian) measurement invariance test: a cross-country illustration with a scale to measure 19 human values. Front Psychol 5:1–10

  • Curran PJ, West SG, Finch JF (1996) The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychol Methods 1:16–29

    Article  Google Scholar 

  • Curran PJ, Bollen KA, Paxton P, Kirby J, Chen F (2002) The noncentral chi-square distribution in misspecified structural equation models: finite sample results from a Monte Carlo simulation. Multivar Behav Res 37(1):1–36

    Article  Google Scholar 

  • Davidov E (2008) A cross-country and cross-time comparison of the human values measurements with the second round of the European Social Survey. Surv Res Methods 2(1):33–46

    Google Scholar 

  • De Jong MG, Steenkamp JBEM, Fox J-P (2007) Relaxing measurement invariance in cross-national consumer research using a hierarchical IRT model. J Consum Res 34(2):260–278

    Article  Google Scholar 

  • DiStefano C (2002) The impact of categorization with confirmatory factor analysis. Struct Equ Model Multidiscip J 9(3):327–346

    Article  MathSciNet  Google Scholar 

  • Dolan CV (1994) Factor analysis of variables with 2, 3, 5, and 7 response categories: a comparison of categorical variable estimators using simulated data. Br J Math Stat Psychol 47:309–326

    Article  Google Scholar 

  • Finch H (2005) The MIMIC Model as a Method for Detecting DIF: Comparison With Mantel-Haenszel, SIBTEST, and the IRT Likelihood Ratio. Appl Psychol Meas 29(4):278–295

    Article  MathSciNet  Google Scholar 

  • Finch WH, French BF (2008) Using exploratory factor analysis for locating invariant referents in factor invariance studies. J Mod Appl Stat Methods 7(1):223–233

    Article  Google Scholar 

  • Flora DB, Curran PJ (2004) An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychol Methods 9(4):466–491

    Article  Google Scholar 

  • Glockner-Rist A, Hoijtink H (2003) The best of both worlds: factor analysis of dichotomous data using item response theory and structural equation modeling. Struct Equ Model Multidiscip J 10(4):544–565

    Article  MathSciNet  Google Scholar 

  • Gonzalez EJ (2012) Rescaling sampling weights and selecting mini-samples from large-scale assessment databases. IERI Monogr Ser Issues Methodol Large Scale Assess 5:117–134

    Google Scholar 

  • Gregorich SE (2006) Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Med Care 44(11):S78–S94

    Article  Google Scholar 

  • Horn JL, McArdle JJ (1992) A practical and theoretical guide to measurement invariance in aging research. Exp Aging Res 18(3):117–144

    Article  Google Scholar 

  • Hu L-T, Bentler PM (1998) Fit indices in covariance structure modeling: sensitivity to underparameterized model misspecification. Psychol Methods 3(4):424–453

    Article  Google Scholar 

  • Jöreskog KG (1971) Simultaneous factor analysis in several populations. Psychometrika 36(4):409–426

    Article  Google Scholar 

  • Kline R (2011) Principles and Practice of Structural Equation Modeling, 3rd edn. The Guilford Press, New York

  • Little TD (1997) Mean and covariance structures (MACS) analyses of cross-cultural data: practical and theoretical issues. Multivar Behav Res 32(1):53–76

    Article  Google Scholar 

  • Little TD, Slegers DW (2005) Factor analysis: multiple groups. In: Everitt BS, Howell DC (eds) Encyclopedia of statistics in behavioral science, vol 2. Wiley, Chichester, pp 617–623

    Google Scholar 

  • Lubke G, Muthén B (2004) Applying multigroup confirmatory factor models for continuous outcomes to Likert scale data complicates meaningful group comparisons. Struct Equ Model 11(4):514–534

    Article  MathSciNet  Google Scholar 

  • MacCallum RC, Browne MW, Sugawara HM (1996) Power analysis and determination of sample size for covariance structure modeling. Psychol Methods 1(2):130–149

    Article  Google Scholar 

  • Maria EO, von Davier M (2014) Toward increasing fairness in score scale calibrations employed in international large-scale assessments. Int J Test 14(1):1–21

  • Meade AW (2005) Sample size and tests of measurement invariance. In: Annual Conference of the Society for Industrial and Organizational Psychology, Los Angeles, CA

  • Meredith W (1993) Measurement invariance, factor analysis and factorial invariance. Psychometrika 58(4):525–543

    Article  MathSciNet  Google Scholar 

  • Meredith W, Millsap RE (1992) On the misuse of manifest variables in the detection of measurement bias. Psychometrika 57(2):289–311

    Article  MathSciNet  Google Scholar 

  • Millsap RE (1995) Measurement invariance, predictive invariance, and the duality paradox. Multivar Behav Res 30(4):577–605

    Article  Google Scholar 

  • Millsap RE (2011) Statistical approaches to measurement invariance. Psychology Press, New York

  • Millsap RE, Olivera-Aguilar M (2012) Investigating measurement invariance using confirmatory factor analysis. The Guilford Press, New York

    Google Scholar 

  • Millsap RE, Yun-Tein J (2004) Assessing factorial invariance in ordered-categorical measures. Multivar Behav Res 39(3):479–515

    Article  Google Scholar 

  • Muthén B, Asparouhov T (2002) Latent variable analysis with categorical outcomes: multiple-group and growth modeling in Mplus. Mplus Web Notes 4. https://www.statmodel.com/examples/webnote.shtml

  • Muthén B, Asparouhov T (2013) BSEM measurement invariance analysis. MPlus Web Notes 17. http://www.statmodel.com/examples/webnote.shtml

  • Muthén B, du Toit SHC, Spisic D (1997) Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. http://pages.gseis.ucla.edu/faculty/muthen/articles/Article_075.pdf

  • Muthén B, Kaplan D (1985) A comparison of some methodologies for the factor analysis of non-normal Likert variables. Br J Math Stat Psychol 38(2):171–189

    Article  Google Scholar 

  • Muthén LK, Muthén BO (1998–2012) Mplus statistical analysis with latent variables: User’s guide, 7th edn. Muthén & Muthén, Los Angeles

  • Organisation for Economic Co-operation and Development (OECD) (2010) TALIS 2008 technical report. OECD Publishing, Paris. http://www.sourceoecd.org/education/97892640798544

  • Preacher KJ, Cai L, MacCallum RC (2007) Alternatives to traditional model comparison strategies for covariance structure models. In: Little TD, Bovaird JA, Card NA (eds) Modeling contextual effects in longitudinal studies, vol 8. Lawrence Erlbaum Associates, Mahwah, pp 33–62

  • Raykov T (1997) Estimation of composite reliability for congeneric measures. Appl Psychol Meas 21:173–184

    Article  Google Scholar 

  • Raykov T, Widaman K (1995) Issues in applied structural equation modeling research

    Article  Google Scholar 

  • Rutkowski L, Svetina D (2013) Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educ Psychol Meas 7(1):31–57

    Article  Google Scholar 

  • Sörbom D (1974) A general method for studying differences in factor means and factor structures between groups. Br J Math Stat Psychol 27(2):229–239

    Article  MathSciNet  Google Scholar 

  • Steenkamp J-BEM, Baumgartner H (1998) Assessing measurement invariance in cross-national consumer research. J Consum Res 25(1):78–107

    Article  Google Scholar 

  • Steiger JH (1990) Structural model evaluation and modification: an interval estimation approach. Multivar Behav Res 25(2):173–180

    Article  Google Scholar 

  • van de Schoot R, Kluytmans A, Tummers L, Lugtig P, Hox J, Muthén B (2013) Facing off with Scylla and Charybdis: a comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Front Psychol 4(770):1–15

    Google Scholar 

  • Vieluf S, Kunter M, van de Vijer FJR (2013) Teacher self-efficacy in cross-national perspective. Teach Teach Educ 35(October):95–103

    Google Scholar 

  • Woods CM, Oltmanns TF (2009) Illustration of MIMIC-Model DIF testing with the schedule for nonadaptive and adaptive personality. J Psychopathol Behav Assess 31(4):320–330

    Article  Google Scholar 

  • Wu AD, Li Z, Zumbo BD (2007) Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: a demonstration with TIMSS data. Pract Assess Res Eval 12(3):1–26

    Google Scholar 

  • Yu C-Y (2002) Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. University of California, Los Angeles

    Google Scholar 

Download references

Acknowledgments

The author would like to acknowledge Steffen Knoll, and the other members of the Research and Analysis Unit (RandA) at the IEA Data Processing and Research Center, Hamburg, Germany for the work and preparation of this study. The author would like to thank Ralph Carstens and Mark Cockle from the IEA DPC for helpful reviews and comments on the previous versions of this paper.

Author information

Authors and Affiliations

  1. Research and Analysis Unit, IEA Data Processing and Research Center, Mexikoring 37, 22297, Hamburg, Germany

    Deana Desa

Authors
  1. Deana Desa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deana Desa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Desa, D. Understanding non-linear modeling of measurement invariance in heterogeneous populations. Adv Data Anal Classif 12, 841–865 (2018). https://doi.org/10.1007/s11634-016-0240-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11634-016-0240-3

Keywords

Mathematics Subject Classification

Search

Navigation