Psychometrika

, 74:49 | Cite as

Reliability of a Longitudinal Sequence of Scale Ratings

  • Annouschka Laenen
  • Ariel Alonso
  • Geert Molenberghs
  • Tony Vangeneugden
Theory and Methods
First Online:
Received:
Revised:

Abstract

Reliability captures the influence of error on a measurement and, in the classical setting, is defined as one minus the ratio of the error variance to the total variance. Laenen, Alonso, and Molenberghs (Psychometrika 73:443–448, 2007) proposed an axiomatic definition of reliability and introduced the R T coefficient, a measure of reliability extending the classical approach to a more general longitudinal scenario. The R T coefficient can be interpreted as the average reliability over different time points and can also be calculated for each time point separately. In this paper, we introduce a new and complementary measure, the so-called R Λ , which implies a new way of thinking about reliability. In a longitudinal context, each measurement brings additional knowledge and leads to more reliable information. The R Λ captures this intuitive idea and expresses the reliability of the entire longitudinal sequence, in contrast to an average or occasion-specific measure. We study the measure’s properties using both theoretical arguments and simulations, establish its connections with previous proposals, and elucidate its performance in a real case study.

Keywords

reliability linear mixed model longitudinal data psychiatry rating scale 
This is a preview of subscription content, log in to check access.

References

  1. Alonso, A., Geys, H., Molenberghs, G., & Vangeneugden, T. (2002). Investigating the criterion validity of psychiatric symptom scales using surrogate marker validation methodology. Journal of Biopharmaceutical Statistics, 12, 161–179. CrossRefPubMedGoogle Scholar
  2. Alonso, A., Geys, H., Molenberghs, G., & Kenward, M.G. (2004). Validation of surrogate markers in multiple randomized clinical trials with repeated measurements: canonical correlation approach. Biometrics, 60, 845–853. CrossRefPubMedGoogle Scholar
  3. Bost, J.E. (1995). The effect of correlated errors on generalizability and dependability coefficients. Applied Psychological Measurement, 19(2), 191–203. CrossRefGoogle Scholar
  4. Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322. Google Scholar
  5. Cole, D.A., Martin, N.C., & Steiger, J.H. (2005). Empirical and conceptual problems with longitudinal trait-state models: introducing a trait-state-occasion model. Psychological Methods, 10(1), 3–20. CrossRefPubMedGoogle Scholar
  6. Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley. Google Scholar
  7. Diggle, P.J., Liang, K.-Y., & Zeger, S.L. (1994). Analysis of longitudinal data. Oxford science publications. Oxford: Clarendon Press. Google Scholar
  8. Heise, D.R. (1969). Separating reliability and stability in test-retest correlation. American Sociological Review, 34, 93–101. CrossRefGoogle Scholar
  9. Hertzog, C., & Nesselroade, J.R. (1987). Beyond autoregressive models: some implications of the trait-state distinction for the structural modeling of developmental change. Child Development, 58, 93–109. CrossRefPubMedGoogle Scholar
  10. Jagodzinski, W., & Kühnel, S.M. (1987). Estimation of reliability and stability in single-indicator multiple-wave models. Sociological Methods and Research, 15, 219–258. CrossRefGoogle Scholar
  11. Johnson, R.A., & Wichern, D.W. (1998). Applied multivariate statistical analysis (4th ed.). Englewood Cliffs: Prentice-Hall. Google Scholar
  12. Kenny, D.A., & Zautra, A. (1995). The trait-state-error model for multiwave data. Journal of Consulting and Clinical Psychology, 63(1), 52–59. CrossRefPubMedGoogle Scholar
  13. Laenen, A., Alonso, A., & Molenberghs, G. (2007). A measure for the reliability of a rating scale based on longitudinal clinical trial data. Psychometrika, 73, 443–448. CrossRefGoogle Scholar
  14. Laenen, A., Alonso, A., Molenberghs, G., & Vangeneugden, T. (2009). A family of parameters to investigate the reliability of a psychiatric symptom scale. Journal of the Royal Statistical Society, Series A, 172, 1–17. Google Scholar
  15. Liang, K.-Y., & Zeger, S.L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22. CrossRefGoogle Scholar
  16. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley. Google Scholar
  17. Molenberghs, G., & Kenward, M.G. (2007). Missing data in clinical studies. Chichester: Wiley. CrossRefGoogle Scholar
  18. Peuskens, J., & the Risperidone Study Group (1995). Risperidone in the treatment of chronic schizophrenic patients: a multinational, multicentre, double-blind, parallel-group study versus haloperidol. British Journal of Psychiatry, 166, 712–726. CrossRefPubMedGoogle Scholar
  19. Raykov, T. (2000). A method for examining stability in reliability. Multivariate Behavioral Research, 35(3), 289–305. CrossRefGoogle Scholar
  20. Royston, P., & Atman, D.G. (1994). Regression using fractional polynomials of continuous covariates: parametric modelling. Applied Statistics, 43(3), 429–467. CrossRefGoogle Scholar
  21. Rubin, D.B. (1976). Inference and missing data. Biometrika, 63, 581–592. CrossRefGoogle Scholar
  22. Searle, S.R. (1982). Matrix algebra useful for statistics. New York: Wiley. Google Scholar
  23. Smith, P.L., & Luecht, R.M. (1992). Correlated effects in generalizability studies. Applied Psychological Measurement, 16(3), 229–235. CrossRefGoogle Scholar
  24. Spearman, C. (1910). Correlation calculate from faulty data. British Journal of Psychology, 3, 271–295. Google Scholar
  25. Tisak, J., & Tisak, M.S. (1996). Longitudinal models of reliability and validity: a latent curve approach. Applied Psychological Measurement, 20, 275–288. CrossRefGoogle Scholar
  26. Vangeneugden, T., Laenen, A., Geys, H., Renard, D., & Molenberghs, G. (2004). Applying linear mixed models to estimate reliability in clinical trial data with repeated measurements. Controlled Clinical Trials, 25, 13–30. CrossRefPubMedGoogle Scholar
  27. Verbeke, G., & Molenberghs, G. (2000). Linear mixed models for longitudinal data. New York: Springer. Google Scholar
  28. Verbyla, A.P., Cullis, B.R., Kenward, M.G., & Welham, S.J. (1999). The analysis of designed experiments and longitudinal data by using smoothing splines. Applied Statistics, 48, 269–311. Google Scholar
  29. Werts, C.E., Linn, C.E., & Jøreskog, K.G. (1977). A simplex model for analyzing academic growth. Educational and Psychological Measurement, 37(3), 745–756. CrossRefGoogle Scholar
  30. Werts, C.E., Breland, H.M., Grandy, J., & Rock, D.R. (1980). Using longitudinal data to estimate reliability in the presence of correlated measurement errors. Educational and Psychological Measurement, 40, 19–29. CrossRefGoogle Scholar
  31. Wiley, D.E., & Wiley, J.A. (1970). The estimation of measurement error in panel data. American Sociological Review, 35, 112–117. CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2008

Authors and Affiliations

  • Annouschka Laenen
    • 1
  • Ariel Alonso
    • 1
  • Geert Molenberghs
    • 1
  • Tony Vangeneugden
    • 2
  1. 1.Hasselt UniversityHasseltBelgium
  2. 2.Tibotec, Johnson & JohnsonHasseltBelgium

Personalised recommendations