Advertisement

Social Indicators Research

, Volume 102, Issue 3, pp 443–461 | Cite as

Assessing the Unidimensionality of Psychological Scales: Using Multiple Criteria from Factor Analysis

  • Suzanne L. Slocum-Gori
  • Bruno D. Zumbo
Article

Abstract

Whenever one uses a composite scale score from item responses, one is tacitly assuming that the scale is dominantly unidimensional. Investigating the unidimensionality of item response data is an essential component of construct validity. Yet, there is no universally accepted technique or set of rules to determine the number of factors to retain when assessing the dimensionality of item response data. Typically factor analysis is used with the eigenvalues-greater-than-one rule, the ratio of first-to-second eigenvalues, parallel analysis, root-mean-square-error-of-approximation, or hypothesis testing approaches involving chi-square tests from Maximum Likelihood or Generalized Least Squares estimation. The purpose of this study was to investigate how these various procedures perform individually, and in combination, when assessing the unidimensionality of item response data via a computer simulated design. Conditions such as sample size, magnitude of communality, distribution of item responses, proportion of communality on second factor, and the number of items with non-zero loadings on the second factor were varied. Results indicate that there was no one individual decision-making method that identified unidimensionality under all conditions manipulated. Given the low communalities, all individual decision-making methods failed to detect unidimensionality for the combination where sample size was small, magnitude of communality was low, and item distributions were skewed. A set of guidelines and a new statistical methodology are provided for researchers.

Keywords

Psychometric testing Construct validity Quality of Life Test dimensionality Rational criteria combinations and indices Unidimensionality methodology 

References

  1. American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1999). Standards for education and psychological testing. Washington, DC: American Psychological Association.Google Scholar
  2. Aron, A., & Aron, E. (2002). Statistics for psychology (2nd ed.). Upper Saddle River, NJ: Prentice Hall.Google Scholar
  3. Boyd, K. C., & Gorsuch, R. L. (2003). Factor replication, factor invariance, and salient loadings: Three objective criteria for number of factors. Manuscript submitted for publication.Google Scholar
  4. Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21, 230–258.CrossRefGoogle Scholar
  5. Bryne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  7. Crawford, C. B. (1975). Determining the number of interpretable factors. Psychological Bulletin, 82, 226–237.CrossRefGoogle Scholar
  8. Crawford, C. B., & Koopman, P. (1973). A note on Horn’s test for the number of factors in factor in analysis. Multivariate Behavioral Research, 8, 117–125.CrossRefGoogle Scholar
  9. Davison, M. L., & Sireci, S. G. (2000). Multidimensional scaling. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematics modeling (pp. 323–352). New York: Academic Press.CrossRefGoogle Scholar
  10. DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327–346.CrossRefGoogle Scholar
  11. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.CrossRefGoogle Scholar
  12. Gessaroli, M. E., & De Champlain, A. F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying responses to a set of items. Journal of Educational Measurement, 33, 157–179.CrossRefGoogle Scholar
  13. Glenberg, A. (1996). Learning from data: An introduction to statistical reasoning. New York, NY: Lawrence Erlbaum Associates.Google Scholar
  14. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  15. Guion, R. M. (1977). Content validity: The source of my discontent. Applied Psychological Measurement, 1, 1–10.CrossRefGoogle Scholar
  16. Hattie, J. (1984). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 20, 1–14.CrossRefGoogle Scholar
  17. Hattie, J. (1985). An empirical study of the various indices for determining unidimensionality. Multivariate Behavioral Research, 19, 49–78.CrossRefGoogle Scholar
  18. Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. The Journal of General Psychology, 123(3), 207–215.CrossRefGoogle Scholar
  19. Humphreys, L. G. (1952). Individual differences. Annual Review of Psychology, 3, 131–150.CrossRefGoogle Scholar
  20. Humphreys, L. G. (1962). The organization of human abilities. American Psychologist, 17, 475–483.CrossRefGoogle Scholar
  21. Kane, M. (2006). Validation. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Washington, DC: American Council on Education and National Council on Measurement in Education.Google Scholar
  22. Lissitz, R. W. (Ed.). (2009). The concept of validity: Revisions, new directions and applications. Charlotte: Information Age Publishing.Google Scholar
  23. Lord, F. M. (1980). Applications of item response theory to practical testing problems. New Jersey: Lawrence Erlbaum.Google Scholar
  24. MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637.CrossRefGoogle Scholar
  25. MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99.CrossRefGoogle Scholar
  26. Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955–966.CrossRefGoogle Scholar
  27. Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 33–45). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  28. Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis. Thousand Oaks, CA: Sage.Google Scholar
  29. Preacher, K. J., & MacCallum, R. C. (2002). Exploratory factor analysis in behavior genetics research: Factor recovery with small sample sizes. Behavior Genetics, 32(2), 13–43.CrossRefGoogle Scholar
  30. Russell, D. W. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Personality and Social Psychology Bulletin, 28(2), 1629–1646.CrossRefGoogle Scholar
  31. Schonemann, P. H. (1981). Power as a function of communality in factor analysis. Bulletin of the Psychonomic Society, 17, 57–60.Google Scholar
  32. Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.Google Scholar
  33. Slocum-Gori, S. L., Zumbo, B. D., Michalos, A. C., & Diener, E. (2009). A note on the dimensionality of quality of life scales: An illustration with the Satisfaction With Life Scale (SWLS). Social Indicators Research: An International Interdisciplinary Journal for Quality of Life Measurement, 92, 489–496.Google Scholar
  34. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589–617.CrossRefGoogle Scholar
  35. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins College Publishers.Google Scholar
  36. Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: An historical overview and some guidelines. Educational and Psychological Measurement, 56, 213–224.Google Scholar
  37. Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 45–79). The Netherlands: Elsevier Science B.V.Google Scholar
  38. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte: Information Age Publishing.Google Scholar
  39. Zumbo, B. D., Gelin, M. N., & Hubley, A. M. (2002). The construction and use of psychological tests and measures. In the Psychology theme of the Encyclopedia of Life Support Systems (EOLSS). Oxford, UK: Eolss Publishers. http://www.educ.ubc.ca/faculty/zumbo/talks/Zumbo_Gelin_Hubley_EOLSS.pdf.

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  1. 1.School of Population and Public Health, Faculty of MedicineUniversity of British ColumbiaVancouverCanada
  2. 2.Measurement, Evaluation, & Research Methodology Program, Department of StatisticsUniversity of British ColumbiaVancouverCanada

Personalised recommendations