Abstract
In this study, we compared the numerical performance of reliability coefficients based on classical test theory and factor analysis. We investigated the coefficients’ divergence from reliability and their population values using unidimensional and multidimensional data generated from both an item response theory and a factor model. In addition, we studied reliability coefficients’ performance when the tested model was misspecified. For unidimensionality, coefficients α, λ2, and coefficient ωu approximated reliability well and were almost unbiased regardless of the data-generating model. For multidimensionality, coefficient ωt performed best with both data generating models. When the tested model was unidimensional but the data multidimensional, all coefficients underestimated reliability. When the tested model incorrectly assumed a common factor in addition to group factors but the data was purely multidimensional, coefficients ωh and ωt identified the underlying data structure well. In practice, we recommend researchers use reliability coefficients that are based on factor analysis when data are multidimensional; when data are unidimensional both classical test theory methods and factor analysis methods get the job done.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bollen, K. A. (1989). Structural equations with latent variables. John Wiley & Sons, Inc.. https://doi.org/10.1002/9781118619179
Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
Holland, P. W. (1990). On the sampling theory foundations of item response theory models. Psychometrika, 55(4), 577–601. https://doi.org/10.1007/BF02294609
Holland, P. W., & Hoskens, M. (2003). Classical test theory as a first-order item response theory: Application to true-score prediction from a possibly nonparallel test. Psychometrika, 68(1), 123–149. https://doi.org/10.1007/BF02296657
Hunt, T. D., & Bentler, P. M. (2015). Quantile lower bounds to reliability based on locally optimal splits. Psychometrika, 80(1), 182–195. https://doi.org/10.1007/s11336-013-9393-6
Jöreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36(2), 109–133. https://doi.org/10.1007/BF02291393
Kim, S. (2012). A note on the reliability coefficients for item response model-based ability estimates. Psychometrika, 77(1), 153–162. https://doi.org/10.1007/s11336-011-9238-0
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
McDonald, R. P. (1999). Test theory: A unified treatment (1st ed.). Psychology Press. https://doi.org/10.4324/9781410601087
Oosterwijk, P. R., Van der Ark, L. A., & Sijtsma, K. (2017). Overestimation of reliability by Guttman’s λ4, λ5, and λ6 and the greatest lower bound. In L. A. van der Ark, S. Culpepper, J. A. Douglas, W.-C. Wang, & M. Wiberg (Eds.), Quantitative psychology research: The 81th annual meeting of the psychometric society 2016 (pp. 159–172). Springer. https://doi.org/10.1007/978-3-319-56294-0_15
Pfadt, J. M., van den Bergh, D., Moshagen, M. (2021a). The reliability of multidimensional scales: A comparison of confidence intervals and a Bayesian alternative [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/d3gfs
Pfadt, J. M., van den Bergh, D., Sijtsma, K., Moshagen, M., & Wagenmakers, E. -J. (2021b). Bayesian estimation of single-test reliability coefficients. Multivariate Behavioral Research. https://doi.org/10.1080/00273171.2021.1891855
Samejima, F. (1968). Estimation of latent ability using a response pattern of graded scores. Educational Testing Service.
Schmid, J., & Leiman, J. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22(1), 53–61. https://doi.org/10.1007/BF02289209
Sijtsma, K., & Pfadt, J. M. (2021). Part II: On the use, the misuse, and the very limited usefulness of Cronbach’s alpha: Discussing lower bounds and correlated errors. Psychometrika, 86(4), 843–860. https://doi.org/10.1007/s11336-021-09789-8
Sijtsma, K., & Van der Ark, L. A. (2021). Measurement models for psychological attributes (1st ed.). Chapman and Hall/CRC. https://doi.org/10.1201/9780429112447
Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. https://doi.org/10.2307/1412159
Woodhouse, B., & Jackson, P. H. (1977). Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: II: A search procedure to locate the greatest lower bound. Psychometrika, 42(4), 579–591. https://doi.org/10.1007/bf02295980
Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s alpha, Revelle’s beta, and McDonald’s omega h: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7
Zinbarg, R. E., Yovel, I., Revelle, W., & McDonald, R. P. (2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for omega h. Applied Psychological Measurement, 30(2), 121–144. https://doi.org/10.1177/0146621605278814
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pfadt, J.M., Sijtsma, K. (2022). Statistical Properties of Lower Bounds and Factor Analysis Methods for Reliability Estimation. In: Wiberg, M., Molenaar, D., González, J., Kim, JS., Hwang, H. (eds) Quantitative Psychology. IMPS 2021. Springer Proceedings in Mathematics & Statistics, vol 393. Springer, Cham. https://doi.org/10.1007/978-3-031-04572-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-04572-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04571-4
Online ISBN: 978-3-031-04572-1
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)