Skip to main content
SpringerLink
Log in

Resampling-Based Inference Methods for Comparing Two Coefficients Alpha

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

The two-sample problem for Cronbach’s coefficient \(\alpha _C\), as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients \(\alpha _{C,1}\) and \(\alpha _{C,2}\). The null hypothesis of interest is \(H_0 : \alpha _{C,1} = \alpha _{C,2}\), which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. \(\alpha \) without any indices denotes the significance level of the corresponding statistical test, not Cronbach’s alpha.

References

  • Bader, K., Hänny, C., Schäfer, V., Neuckel, A., & Kuhl, C. (2009). Childhood Trauma Questionnaire—Psychometrische Eigenschaften einer deutschsprachigen Version. Zeitschrift für Klinische Psychologie und Psychotherapie, 38(4), 223–230.

    Article  Google Scholar 

  • Bentler, P. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bernstein, D. P., et al. (2003). Development and validation of a brief screening version of the childhood trauma questionnaire. Child Abuse & Neglect, 27(2), 169–190.

    Article  Google Scholar 

  • Bonett, D. G. (2003). Sample size requirements for comparing two alpha coefficients. Applied Psychological Measurement, 27(1), 72–74.

    Article  Google Scholar 

  • Bonett, D. G., & Wright, T. A. (2015). Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning. Journal of Organizational Behavior, 36(1), 3–15.

    Article  Google Scholar 

  • Bradbury, I. (1987). Analysis of variance versus randomization tests—A comparison. British Journal of Mathematical and Statistical Psychology, 40(2), 177–187.

    Article  Google Scholar 

  • Brennan, R. L. (2001). Generalizability Theory. New York: Springer.

    Book  Google Scholar 

  • Chung, E., & Romano, J. P. (2013). Exact and asymptotically robust permutation tests. The Annals of Statistics, 41(2), 484–507.

    Article  Google Scholar 

  • Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98.

    Article  Google Scholar 

  • Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Princeton: ERIC.

    Google Scholar 

  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.

    Article  Google Scholar 

  • D’Zurilla, T. J., Nezu, A. M., & Maydeu-Olivares, A. (2002). Social problem-solving inventory—Revised (SPSI-R). North Tonawanda: Multi Health Systems.

    Google Scholar 

  • Furr, R. M., & Bacharach, V. R. (2013). Psychometrics: An introduction. Thousand Oaks: Sage.

    Google Scholar 

  • Green, S. B., & Yang, Y. (2009a). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74(1), 121–135.

    Article  Google Scholar 

  • Green, S. B., & Yang, Y. (2009b). Reliability on summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155–167.

    Article  Google Scholar 

  • Gulliksen, H. (2013). Theory of mental tests. Abingdon: Routledge.

    Google Scholar 

  • Guttmann, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255–282.

    Article  Google Scholar 

  • Hogan, T. P., Benjamin, A., & Brezinski, K. L. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60(4), 523–531.

    Article  Google Scholar 

  • Janssen, A. (1997). Studentized permutation tests for non-iid hypotheses and the generalized Behrens–Fisher problem. Statistics & Probability Letters, 36(1), 9–21.

    Article  Google Scholar 

  • Kim, S., & Feldt, L. S. (2008). A comparison of tests for equality of two or more independent alpha coefficients. Journal of Educational Measurement, 45(2), 179–193.

    Article  Google Scholar 

  • Konietschke, F., Bathke, A. C., Harrar, S. W., & Pauly, M. (2015). Parametric and non-parametric bootstrap methods for general MANOVA. Journal of Multivariate Analysis, 140, 291–301.

    Article  Google Scholar 

  • Konietschke, F., & Pauly, M. (2014). Bootstrapping and permuting paired t-test type statistics. Statistics and Computing, 24(3), 283–296.

    Article  Google Scholar 

  • Kuijpers, R. E., Ark, L. A., & Croon, M. A. (2013). Testing hypotheses involving Cronbach’s alpha using marginal models. British Journal of Mathematical and Statistical Psychology, 66(3), 503–520.

    PubMed  Google Scholar 

  • Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley.

    Google Scholar 

  • Maydeu-Olivares, A., Coffman, D. L., & Hartmann, W. M. (2007). Asymptotically distribution-free (ADF) interval estimation of coefficient alpha. Psychological Methods, 12(2), 157.

    Article  PubMed  Google Scholar 

  • Maydeu-Olivares, A., Coffman, D. L., García-Forero, C., & Gallardo-Pujol, D. (2010). Hypothesis testing for coefficient alpha: An SEM approach. Behavior research methods, 42(2), 618–625.

    Article  PubMed  PubMed Central  Google Scholar 

  • McDonald, R. P. (1978). Generalizability in factorable domains: domain validity and generalizability 1. Educational and Psychological Measurement, 38(1), 75–79.

    Article  Google Scholar 

  • McDonald, R. P. (1999). Test theory: A unified treatment. New York: Psychology Press.

    Google Scholar 

  • Mellenbergh, G. J. (1996). Measurement precision in test score and item response models. Psychological Methods, 1(3), 293.

    Article  Google Scholar 

  • Molenaar, I. W., & Sijtsma, K. (1988). Mokken’s approach to reliability estimation extended to multicategory items. Kwantitatieve Methoden, 9(28), 115–126.

    Google Scholar 

  • Muthén, B., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189.

    Article  Google Scholar 

  • Muthén, B., & Kaplan, D. (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model. British Journal of Mathematical and Statistical Psychology, 45(1), 19–30.

    Article  Google Scholar 

  • Novick, M. R., & Lewis, C. (1966). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32(1), 1–13.

    Article  Google Scholar 

  • Nunnally, J. C., & Bernstein, I. H. (1978). Psychometric theory. New York: McGraw-Hill.

    Google Scholar 

  • Omelka, M., & Pauly, M. (2012). Testing equality of correlation coefficients in two populations via permutation methods. Journal of Statistical Planning and Inference, 142(6), 1396–1406.

    Article  Google Scholar 

  • Padilla, M. A., Divers, J., & Newton, M. (2012). Coefficient alpha bootstrap confidence interval under nonnormality. Applied Psychological Measurement, 36(5), 331–348.

    Article  Google Scholar 

  • Pauly, M. (2011). Discussion about the quality of F-ratio resampling tests for comparing variances. Test, 20(1), 163–179.

    Article  Google Scholar 

  • Pauly, M., Brunner, E., & Konietschke, F. (2015). Asymptotic permutation tests in general factorial designs. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 77(2), 461–473.

    Article  Google Scholar 

  • Peterson, R. A. (1994). A meta-analysis of Cronbach’s coefficient alpha. Journal of Consumer Research, 21(2), 381–391.

    Article  Google Scholar 

  • Prelog, A. J., Berry, K. J., & Mielke, P. W, Jr. (2009). Resampling permutation probability values for Cronbach’s alpha. Perceptual and Motor Skills, 108(2), 431–438.

    Article  PubMed  Google Scholar 

  • R Core Team (2016). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.

  • Rao, C. R., & Sinharay, S. (2006). Handbook of statistics: Psychometrics (Vol. 26). New York: Elsevier.

    Google Scholar 

  • Revelle, W., & Zinbarg, R. E. (2009). Coefficient alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145–154.

    Article  Google Scholar 

  • Schury, K., Zimmermann, J., Umlauft, M., Hulbert, A. L., Guendel, H., Ziegenhain, U., et al. (2017). Childhood maltreatment, postnatal distress and the protective role of social support. Child Abuse & Neglect, 67, 228–239.

    Article  Google Scholar 

  • Sijtsma, K. (2009a). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120.

    Article  PubMed  Google Scholar 

  • Sijtsma, K. (2009b). Reliability beyond theory and into practice. Psychometrika, 74(1), 169–173.

    Article  PubMed  Google Scholar 

  • Sijtsma, K., & Molenaar, I. W. (1987). Reliability of test score in nonparametric item response theory. Psychometrika, 52, 79–97.

    Article  Google Scholar 

  • Still, A., & White, A. (1981). The approximate randomization test as an alternative to the F-test in analysis of variance. British Journal of Mathematical and Statistical Psychology, 34(2), 243–252.

    Article  Google Scholar 

  • Ten Berge, J. M., & Soc̆an, G. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613–625.

    Article  Google Scholar 

  • Umlauft, M., Konietschke, F., & Pauly, M. (2017). Rank-based permutation approaches for nonparametric factorial designs. British Journal of Mathematical and Statistical Psychology, 70(3), 368–390.

    Article  PubMed  Google Scholar 

  • van der Ark, L. A. (2012). New developments in Mokken scale analysis in R. Journal of Statistical Software, 48(5), 1–27.

    Article  Google Scholar 

  • van der Ark, L. A., van der Palm, D. W., & Sijtsma, K. (2011). A latent class approach to estimating test-score reliability. Applied Psychological Measurement, 35, 380–392.

    Article  Google Scholar 

  • van Zyl, J. M., Neudecker, H., & Nel, D. (2000). On the distribution of the maximum likelihood estimator of Cronbach’s alpha. Psychometrika, 65(3), 271–280.

    Article  Google Scholar 

  • Yuan, K.-H., Guarnaccia, C. A., & Hayslip, B. (2003). A study of the distribution of sample coefficient alpha with the Hopkins symptom checklist: Bootstrap versus asymptotics. Educational and Psychological Measurement, 63(1), 5–23.

    Article  Google Scholar 

  • Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbachs \(\alpha \), Revelles \(\beta \), and McDonalds \(\omega _h\): Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Statistics, Ulm University, Ulm, Germany

    Markus Pauly & Maria Umlauft

  2. Technical University of Munich, Munich, Germany

    Ali Ünlü

Authors
  1. Markus Pauly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Umlauft
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ali Ünlü
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Markus Pauly.

Additional information

The work of Markus Pauly and Maria Umlauft was supported by the German Research Foundation project DFG-PA 2409/3-1.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pauly, M., Umlauft, M. & Ünlü, A. Resampling-Based Inference Methods for Comparing Two Coefficients Alpha. Psychometrika 83, 203–222 (2018). https://doi.org/10.1007/s11336-017-9601-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-017-9601-x

Keywords

Search

Navigation