Abstract
Reliability captures the influence of error on a measurement and, in the classical setting, is defined as one minus the ratio of the error variance to the total variance. Laenen, Alonso, and Molenberghs (Psychometrika 73:443–448, 2007) proposed an axiomatic definition of reliability and introduced the R T coefficient, a measure of reliability extending the classical approach to a more general longitudinal scenario. The R T coefficient can be interpreted as the average reliability over different time points and can also be calculated for each time point separately. In this paper, we introduce a new and complementary measure, the so-called R Λ , which implies a new way of thinking about reliability. In a longitudinal context, each measurement brings additional knowledge and leads to more reliable information. The R Λ captures this intuitive idea and expresses the reliability of the entire longitudinal sequence, in contrast to an average or occasion-specific measure. We study the measure’s properties using both theoretical arguments and simulations, establish its connections with previous proposals, and elucidate its performance in a real case study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alonso, A., Geys, H., Molenberghs, G., & Vangeneugden, T. (2002). Investigating the criterion validity of psychiatric symptom scales using surrogate marker validation methodology. Journal of Biopharmaceutical Statistics, 12, 161–179.
Alonso, A., Geys, H., Molenberghs, G., & Kenward, M.G. (2004). Validation of surrogate markers in multiple randomized clinical trials with repeated measurements: canonical correlation approach. Biometrics, 60, 845–853.
Bost, J.E. (1995). The effect of correlated errors on generalizability and dependability coefficients. Applied Psychological Measurement, 19(2), 191–203.
Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322.
Cole, D.A., Martin, N.C., & Steiger, J.H. (2005). Empirical and conceptual problems with longitudinal trait-state models: introducing a trait-state-occasion model. Psychological Methods, 10(1), 3–20.
Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
Diggle, P.J., Liang, K.-Y., & Zeger, S.L. (1994). Analysis of longitudinal data. Oxford science publications. Oxford: Clarendon Press.
Heise, D.R. (1969). Separating reliability and stability in test-retest correlation. American Sociological Review, 34, 93–101.
Hertzog, C., & Nesselroade, J.R. (1987). Beyond autoregressive models: some implications of the trait-state distinction for the structural modeling of developmental change. Child Development, 58, 93–109.
Jagodzinski, W., & Kühnel, S.M. (1987). Estimation of reliability and stability in single-indicator multiple-wave models. Sociological Methods and Research, 15, 219–258.
Johnson, R.A., & Wichern, D.W. (1998). Applied multivariate statistical analysis (4th ed.). Englewood Cliffs: Prentice-Hall.
Kenny, D.A., & Zautra, A. (1995). The trait-state-error model for multiwave data. Journal of Consulting and Clinical Psychology, 63(1), 52–59.
Laenen, A., Alonso, A., & Molenberghs, G. (2007). A measure for the reliability of a rating scale based on longitudinal clinical trial data. Psychometrika, 73, 443–448.
Laenen, A., Alonso, A., Molenberghs, G., & Vangeneugden, T. (2009). A family of parameters to investigate the reliability of a psychiatric symptom scale. Journal of the Royal Statistical Society, Series A, 172, 1–17.
Liang, K.-Y., & Zeger, S.L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley.
Molenberghs, G., & Kenward, M.G. (2007). Missing data in clinical studies. Chichester: Wiley.
Peuskens, J., & the Risperidone Study Group (1995). Risperidone in the treatment of chronic schizophrenic patients: a multinational, multicentre, double-blind, parallel-group study versus haloperidol. British Journal of Psychiatry, 166, 712–726.
Raykov, T. (2000). A method for examining stability in reliability. Multivariate Behavioral Research, 35(3), 289–305.
Royston, P., & Atman, D.G. (1994). Regression using fractional polynomials of continuous covariates: parametric modelling. Applied Statistics, 43(3), 429–467.
Rubin, D.B. (1976). Inference and missing data. Biometrika, 63, 581–592.
Searle, S.R. (1982). Matrix algebra useful for statistics. New York: Wiley.
Smith, P.L., & Luecht, R.M. (1992). Correlated effects in generalizability studies. Applied Psychological Measurement, 16(3), 229–235.
Spearman, C. (1910). Correlation calculate from faulty data. British Journal of Psychology, 3, 271–295.
Tisak, J., & Tisak, M.S. (1996). Longitudinal models of reliability and validity: a latent curve approach. Applied Psychological Measurement, 20, 275–288.
Vangeneugden, T., Laenen, A., Geys, H., Renard, D., & Molenberghs, G. (2004). Applying linear mixed models to estimate reliability in clinical trial data with repeated measurements. Controlled Clinical Trials, 25, 13–30.
Verbeke, G., & Molenberghs, G. (2000). Linear mixed models for longitudinal data. New York: Springer.
Verbyla, A.P., Cullis, B.R., Kenward, M.G., & Welham, S.J. (1999). The analysis of designed experiments and longitudinal data by using smoothing splines. Applied Statistics, 48, 269–311.
Werts, C.E., Linn, C.E., & Jøreskog, K.G. (1977). A simplex model for analyzing academic growth. Educational and Psychological Measurement, 37(3), 745–756.
Werts, C.E., Breland, H.M., Grandy, J., & Rock, D.R. (1980). Using longitudinal data to estimate reliability in the presence of correlated measurement errors. Educational and Psychological Measurement, 40, 19–29.
Wiley, D.E., & Wiley, J.A. (1970). The estimation of measurement error in panel data. American Sociological Review, 35, 112–117.
Author information
Authors and Affiliations
Corresponding author
Additional information
The authors are grateful to J&J PRD for kind permission to use their data. We gratefully acknowledge support from Belgian IUAP/PAI network “Statistical Techniques and Modeling for Complex Substantive Questions with Complex Data.”
Rights and permissions
About this article
Cite this article
Laenen, A., Alonso, A., Molenberghs, G. et al. Reliability of a Longitudinal Sequence of Scale Ratings. Psychometrika 74, 49–64 (2009). https://doi.org/10.1007/s11336-008-9079-7
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-008-9079-7