Skip to main content

Advertisement

SpringerLink
Log in

Strengthening the assessment of factorial invariance across population subgroups: a commentary on Varni et al. (2013)

  • Commentary
  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Objectives

This article provides a commentary in response to “Varni et al. (Qual Life Res. doi:10.1007/s11136-013-0370-4, 2013)."

Methods and results

The commentary argues that the approximate model fit indexes commonly used in maximum-likelihood confirmatory factor analysis and factorial invariance testing are seriously flawed, as they overlook potentially serious model misspecifications that could bias parameter estimates and compromise inference.

Conclusions

Flexible and convenient Bayesian estimation approaches are presented that can substantially aid in: (1) resolving commonly encountered specification errors in confirmatory factor models and (2) locating specific measurement parameters that are non-invariant across population subgroups. It is recommended that these methods should be more widely adopted for evaluating the factorial invariance of patient-reported outcome measures and other types of instruments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Abbreviations

AFI:

Approximate fit index

CFA:

Confirmatory factor analysis

HRQoL:

Health-related quality of life

MCMC:

Markov chain Monte Carlo

SEM:

Structural equation modeling

PedsQL™ MFS:

Pediatric Quality of Life Inventory™ Multidimensional Fatigue Scale

PPP:

Posterior predictive p value

PROM:

Patient-reported outcome measure

References

  1. Doward, L. C., Gnanasakthy, A., & Baker, M.G. (2010). Patient reported outcomes: looking beyond the label claim. Health and Quality of Life Outcomes, 8(89), 1–9.

    Google Scholar 

  2. Dinan, M. A., Compton, K. L., Dhillon, J. K., Hammill, B. G., Dewitt, E. M., Weinfurt, K. P., et al. (2011). Use of patient-reported outcomes in randomized, double-blind, placebo-controlled clinical trials. Medical Care, 49(4), 415–419.

    PubMed  Google Scholar 

  3. Sprangers, M. A. (2010). Disregarding clinical trial-based patient-reported outcomes is unwarranted: Five advances to substantiate the scientific stringency of quality-of-life measurement. Acta Oncologica, 49(2), 155–163.

    Article  PubMed  Google Scholar 

  4. Snyder, C. F., Aaronson, N. K., Choucair, A. K., Elliott, T. E., Greenhalgh, J., Halyard, M. Y., et al. (2012). Implementing patient-reported outcomes assessment in clinical practice: A review of the options and considerations. Quality of Life Research, 21(8), 1305–1314.

    Article  PubMed  Google Scholar 

  5. McKenna, S. P. (2011). Measuring patient-reported outcomes: moving beyond misplaced common sense to hard science. BMC Medicine, 9(86), 1–12.

    Google Scholar 

  6. Jones, P., Miravitlles, M., van der Molen, T., & Kulich, K. (2012). Beyond FEV1 in COPD: A review of patient-reported outcomes and their measurement. International Journal of Chronic Obstructive Pulmonary Disease, 7, 697–709.

    Article  PubMed  Google Scholar 

  7. Swartz, R. J., Schwartz, C., Basch, E., Cai, L., Fairclough, D. L., McLeod, L., et al. (2011). The king’s foot of patient-reported outcomes: Current practices and new developments for the measurement of change. Quality of Life Research, 20(8), 1159–1167.

    Article  PubMed  Google Scholar 

  8. Cook, K. F., Bamer, A. M., Amtmann, D., Molton, I. R., & Jensen, M. P. (2012). Six patient-reported outcome measurement information system short form measures have negligible age- or diagnosis-related differential item functioning in individuals with disabilities. Archives of Physical Medicine and Rehabilitation, 93(7), 1289–1291.

    Article  PubMed  Google Scholar 

  9. Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429.

    Article  PubMed  Google Scholar 

  10. Varni, J. W., Beaujean, A. A., & Limbers, C. A. (2013). Factorial invariance of pediatric patient self-reported fatigue across age and gender: A multigroup confirmatory factor analysis approach utilizing the PedsQL™ Multidimensional Fatigue Scale. Quality of Life Research, Online First. doi:10.1007/s11136-013-0370-4.

  11. Jöreskog, K. G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34, 183–202.

    Article  Google Scholar 

  12. Yuan, K.-H., & Bentler, P. M. (2004). On Chi square difference and z tests in mean and covariance structure analysis when the base model is misspecified. Educational and Psychological Measurement, 64(5), 737–757.

    Article  Google Scholar 

  13. Steiger, J. H., Shapiro, A., & Browne, M. W. (1985). On the multivariate distribution of sequential Chi square statistics. Psychometrika, 50(3), 253–264.

    Article  Google Scholar 

  14. Marsh, H. W., Hau, K.-T., & Grayson, D. (2005). Goodness of fit evaluation in structural equation modeling. In A. Maydeu-Olivares & J. McCardle (Eds.), Contemporary psychometrics: A festschrift to Roderick P. McDonald (pp. 275–340). Mahwah, NJ: Erlbaum.

    Google Scholar 

  15. Bagozzi, R. P., & Yi, Y. (2012). ‘Specification, evaluation, and interpretation of structural equation models. Journal of the Academy of Marketing Science, 40, 8–34.

    Article  Google Scholar 

  16. Preacher, K. J., & Merkle, E. C. (2012). The problem of model selection uncertainty in structural equation modelling. Psychological Methods, 17(1), 1–14.

    Article  PubMed  Google Scholar 

  17. Byrne, B. M. (2008). Testing for multigroup equivalence of a measuring instrument: A walk through the process. Psicothema, 20(4), 872–882.

    PubMed  Google Scholar 

  18. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504.

    Article  Google Scholar 

  19. Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93(3), 568–592.

    Article  PubMed  Google Scholar 

  20. McIntosh, C. N. (2007). Rethinking fit assessment in structural equation modelling: A commentary and elaboration on Barrett (2007). Personality and Individual Differences, 42(5), 859–867.

    Article  Google Scholar 

  21. McIntosh, C. N. (2007). Improving the evaluation of model fit in confirmatory factor analysis: A commentary on Gundy, C. M., Fayers, P. M., Groenvold, M., Petersen, M. Aa., Scott, N. W., Sprangers, M. A. J., Velikov, G., Aaronson, N. K. (2011). Comparing higher-order models for the EORTC QLQ-C30. Quality of Life Research. Quality of Life Research, 21(9), 1619–1621. doi:10.1007/s11136-011-0082-6.

  22. Hayduk, L. A., Cummings, G., Boadu, K., Pazderka-Robinson, H., & Boulianne, S. (2007). Testing! testing! one, two, three—testing the theory in structural equation models! Personality and Individual Differences, 42(5), 841–850.

    Article  Google Scholar 

  23. Hayduk, L. A., & Glaser, D. N. (2000). Jiving the four-step, waltzing around factor analysis, and other serious fun. Structural Equation Modeling, 7(1), 1–35.

    Article  Google Scholar 

  24. Shipley, B. (2002). Cause and correlation in biology: A user’s guide to path analysis, structural equations and causal inference (2nd ed.). Cambridge, UK: Cambridge University Press.

    Google Scholar 

  25. Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York: Guilford Press.

    Google Scholar 

  26. Saris, W. E., Satorra, A., & van der Veld, W. (2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling, 16(4), 561–582.

    Article  Google Scholar 

  27. Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086–1120.

    Article  Google Scholar 

  28. Yuan, K.-H., Kouros, C. D., & Kelley, K. (2008). Diagnosis for covariance structure models by analyzing the path. Structural Equation Modeling, 15, 564–602.

    Article  Google Scholar 

  29. Kolenikov, S. (2011). Biases of parameter estimates in misspecified structural equation models. Sociological Methodology, 41(1), 119–157.

    Article  Google Scholar 

  30. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, New Jersey: Erlbaum.

    Google Scholar 

  31. Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696.

    Article  PubMed  Google Scholar 

  32. Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397–438.

    Article  Google Scholar 

  33. Morin, A. J. S., Marsh, H. W., & Nagengast, B. (in press). Exploratory structural equation modeling. To appear in G.R. Hancock & R.O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed.) Charlotte, NC: Information Age Publishing.

  34. Cole, D. A., Ciesla, J. A., & Steiger, J. H. (2007). The insidious effects of failing to include design-driven correlated residuals in latent variable covariance structure analysis. Psychological Methods, 12(4), 381–398.

    Article  PubMed  Google Scholar 

  35. Saris, W. E., & Aalberts, C. (2003). Different explanations for correlated disturbance terms in MTMM studies. Structural Equation Modeling, 10(2), 193–213.

    Article  Google Scholar 

  36. Reddy, S. K. (1992). Effects of ignoring correlated measurement error in structural equation models. Educational and Psychological Measurement, 52(3), 549–570.

    Article  Google Scholar 

  37. Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244.

    Article  Google Scholar 

  38. MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modification in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490–504.

    Article  PubMed  CAS  Google Scholar 

  39. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.

    Google Scholar 

  40. Muthén, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: a more flexible representation of substantive theory. Psychological Methods, 17(3), 313–335.

    Article  PubMed  Google Scholar 

  41. Muthen, B., & Asparouhov, T. (January 11, 2013). BSEM measurement invariance analysis. Mplus Web Notes: No. 17. Accessed 15 March 2013 at: http://www.statmodel.com/examples/webnotes/webnote17.pdf.

  42. Golay, P., Reverte, I., Rossier, J., Favez, N., & Lecerf, T. (2012). Further insights on the French WISC-IV factor structure through Bayesian structural equation modeling. Psychological Assessment, Online First. doi:10.1037/a0030676.

  43. Andrews, M., & Baguley, T. (2013). Prior approval: The growth of Bayesian methods in psychology. British Journal of Mathematical and Statistical Psychology, 66(1), 1–7.

    Article  PubMed  Google Scholar 

  44. Wetzels, R., & Wagenmakers, E.-J. (2010). Exemplary introduction to Bayesian statistical inference. (book review of “Bayesian modeling using WinBUGS”). Journal of Mathematical Psychology, 54, 466–469.

    Article  Google Scholar 

  45. Poirier, D. J. (2006). The growth of Bayesian methods in statistics and economics since 1970. Bayesian Analysis, 1(4), 969–980.

    Article  Google Scholar 

  46. Alston, C. L., Mengersen, K. L., & Pettitt, A. N. (Eds.). (2013). Case studies in Bayesian statistical modelling and analysis. Chichester, UK: Wiley.

    Google Scholar 

  47. Jackman, S. (2009). Bayesian analysis for the social sciences. West Sussex, UK: Wiley.

    Book  Google Scholar 

  48. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003). Bayesian data analysis (2nd ed.). Boca Raton, FL: Chapman and Hall/CRC Press.

    Google Scholar 

  49. Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008). A weakly informative default prior distribution for logistic and other models. The Annals of Applied Statistics, 2(4), 1360–1383.

    Article  Google Scholar 

  50. Chow, S. M., Tang, N., Yuan, Y., Song, X., & Zhu, H. (2011). Bayesian estimation of semiparametric nonlinear dynamic factor analysis models using the Dirichlet process prior. British Journal of Mathematical and Statistical Psychology, 64(1), 69–106.

    Article  PubMed  Google Scholar 

  51. Kyung, M., Gill, J., & Casella, G. (2011). New findings from terrorism data: Dirichlet process random-effects models for latent groups. Journal of the Royal Statistical Society: Series C (Applied Statistics), 60(5), 701–721.

    Article  Google Scholar 

  52. Brooks, S., Gelman, A., Jones, G., & Meng, X.-L. (Eds.). (2011). Handbook of Markov chain Monte Carlo. Boca Raton, FL: Chapman and Hall/CRC.

    Google Scholar 

  53. Eberly, L. E., & Casella, G. (2003). Estimating Bayesian credible intervals. Journal of Statistical Planning and Inference, 112, 115–132.

    Article  Google Scholar 

  54. Curran, J. M. (2005). An introduction to Bayesian credible intervals for sampling error in DNA profiles. Law, Probability and Risk, 4, 115–126.

    Article  Google Scholar 

  55. Kruschke, J. K., Aguinis, H., & Joo, H. (2012). The time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15, 722–752.

    Article  Google Scholar 

  56. Yuan, Y., & MacKinnon, D. P. (2009). Bayesian mediation analysis. Psychological Methods, 14(4), 301–322.

    Article  PubMed  Google Scholar 

  57. Lee, S.-Y., & Song, X.-Y. (2012). Basic and advanced Bayesian structural equation modeling: with applications in the medical and behavioral sciences. Chichester, UK: Wiley.

    Google Scholar 

  58. Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). San Diego, CA: Academic Press.

    Google Scholar 

  59. Levy, R. (2011). Bayesian data-model fit assessment for structural equation modeling. Structural Equation Modeling, 18(4), 663–685.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Crime Prevention Centre, Public Safety Canada, 269 Laurier Avenue West, Ottawa, ON, K1A 0P8, Canada

    Cameron N. McIntosh

Authors
  1. Cameron N. McIntosh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cameron N. McIntosh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

McIntosh, C.N. Strengthening the assessment of factorial invariance across population subgroups: a commentary on Varni et al. (2013). Qual Life Res 22, 2595–2601 (2013). https://doi.org/10.1007/s11136-013-0465-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-013-0465-y

Keywords

Search

Navigation