Skip to main content
SpringerLink
Log in

Likelihood-Based Clustering of Meta-Analytic SROC Curves

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

Meta-analysis of diagnostic studies experience the common problem that different studies might not be comparable since they have been using a different cut-off value for the continuous or ordered categorical diagnostic test value defining different regions for which the diagnostic test is defined to be positive. Hence specificities and sensitivities arising from different studies might vary just because the underlying cut-off value had been different. To cope with the cut-off value problem interest is usually directed towards the receiver operating characteristic (ROC) curve which consists of pairs of sensitivities and false-positive rates (1-specificity). In the context of meta-analysis one pair represents one study and the associated diagram is called an SROC curve where the S stands for “summary”. In meta-analysis of diagnostic studies emphasis has traditionally been placed on modelling this SROC curve with the intention of providing a summary measure of the diagnostic accuracy by means of an estimate of the summary ROC curve. Here, we focus instead on finding sub-groups or components in the data representing different diagnostic accuracies. The paper will consider modelling SROC curves with the Lehmann family which is characterised by one parameter only. Each single study can be represented by a specific value of that parameter. Hence we focus on the distribution of these parameter estimates and suggest modelling a potential heterogeneous or cluster structure by a mixture of specifically parameterised normal densities. We point out that this mixture is completely nonparametric and the associated mixture likelihood is well-defined and globally bounded. We use the theory and algorithms of nonparametric mixture likelihood estimation to identify a potential cluster structure in the diagnostic accuracies of the collection of studies to be analysed. Several meta-analytic applications on diagnostic studies, including AUDIT and AUDIT-C for detection of unhealthy alcohol use, the mini-mental state examination for cognitive disorders, as well as diagnostic accuracy inspection data on metal fatigue of aircraft spare parts, are discussed to illustrate the methodology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aergeerts, B., Buntinx, F., & Kester, A. (2004). The value of the CAGE in screening for alcohol abuse and alcohol dependence in general clinical populations: A diagnostic meta-analysis. Journal of Clinical Epidemiology, 57, 30–39.

    Article  Google Scholar 

  • Aitkin, M. (1999a). A general maximum likelihood analysis of variance components in generalized linear models. Biometrics, 55, 117–128.

    Article  PubMed  Google Scholar 

  • Aitkin, M. (1999b). Meta-analysis by random effect modelling in generalized linear models. Statistics in Medicine, 18, 2343–2351.

    Article  PubMed  Google Scholar 

  • Biggerstaff, B.J., & Tweedie, R.L. (1997). Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Statistics in Medicine, 16, 753–768.

    Article  PubMed  Google Scholar 

  • Böhning, D., Dietz, E., & Schlattmann, P. (1998). Recent developments in computer-assisted analysis of mixtures (C.A.MAN). Biometrics, 54, 367–377.

    Article  Google Scholar 

  • Böhning, D. (2000). Computer-assisted analysis of mixtures and applications. Meta-analysis, disease mapping and others. Boca Raton: Chapman & Hall/CRC.

    Google Scholar 

  • Böhning, D., Malzahn, U., Dietz, E., Schlattmann, P., Viwatwongkasem, C., & Biggeri, A. (2002). Some general points in estimating heterogeneity variance with the DerSimonian–Laird estimator. Biostatistics, 3, 445–457.

    Article  PubMed  Google Scholar 

  • Böhning, D. (2003). The EM algorithm with gradient function update for discrete mixtures with known (fixed) number of components. Statistics and Computing, 13, 257–265.

    Article  Google Scholar 

  • Böhning, D., & Viwatwongkasem, C. (2005). Revisiting proportion estimators. Statistical Methods in Medical Research, 14, 1–23.

    Article  Google Scholar 

  • Böhning, D., Holling, H., & Böhning, W. (2008). Revisiting Youden’s index as a useful measure of the misclassification error in meta-analysis of diagnostic studies. Statistical Methods in Medical Research, 17, 543–554.

    Article  PubMed  Google Scholar 

  • Brockwell, S.E., & Gordon, I.R. (2001). A comparison of statistical methods for meta-analysis. Statistical Methods in Medical Research, 20, 825–840.

    Google Scholar 

  • Clayton, D.G., & Kaldor, J. (1987). Empirical Bayes estimates of age-standardized relative risks for use in disese mapping. Biometrics, 43, 671–681.

    Article  PubMed  Google Scholar 

  • Cooper, H., & Hedges, L. (1994). The handbook of research synthesis. New York: Russell Sage Foundation.

    Google Scholar 

  • Dempster, A.P., Laird, N.M., & Rubin, D.B. (1977). Maximum likelihood estimation from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society. Series B, 39, 1–38.

    Google Scholar 

  • DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.

    Article  PubMed  Google Scholar 

  • Egger, M., Smith, G.D., & Altman, D.G. (2001). Systematic reviews in health care: Meta-analysis in context. London: BMJ Publishing Group.

    Book  Google Scholar 

  • Gatsonis, C., & Paliwal, P. (2006). Meta-analysis of diagnostic and screening test accuracy evaluations: Methodologic primer. American Journal of Roentgenology, 187, 271–281.

    Article  PubMed  Google Scholar 

  • Harbord, R.M., Deeks, J.J., Egger, M., Whiting, P., & Sterne, J.A.C. (2007). A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics, 8, 239–251.

    Article  PubMed  Google Scholar 

  • Hardy, R.J., & Thompson, S.G. (1996). A likelihood approach to meta-analysis with random effects. Statistics in Medicine, 15, 619–629.

    Article  PubMed  Google Scholar 

  • Hardy, R.J., & Thompson, S.G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17, 841–856.

    Article  PubMed  Google Scholar 

  • Hasselblad, V., & Hedges, L.V. (1995). Meta-analysis of screening and diagnostic tests. Psychological Bulletin, 117, 167–178.

    Article  PubMed  Google Scholar 

  • Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.

    Google Scholar 

  • Hunter, J.E., Schmidt, F.L., & Jackson, B.G. (1982). Meta-analysis: Cumulative research findings across studies. Beverly Hills: Sage Publications.

    Google Scholar 

  • Irwig, L., Tosteson, A.N., Gatsonis, C., Lau, J., Colditz, G., Chalmers, T.C., & Mosteller, F. (1994). Guidelines for meta-analyses evaluating diagnostic tests. Annals of Internal Medicine, 120, 667–676.

    PubMed  Google Scholar 

  • Irwig, L., Macaskill, P., Glasziou, P., & Fahey, M. (1995). Meta-analytic methods for diagnostic test accuracy. Journal of Clinical Epidemiology, 48, 119–130.

    Article  PubMed  Google Scholar 

  • Kiefer, J., & Wolfowitz, J. (1956). Consistency of the maximum likelihood estimator in the presence of infinitely many incidental parameters. Annals of Mathematical Statistics, 27, 886–906.

    Google Scholar 

  • Kriston, L., Hölzel, L., Weiser, A., Berner, M.M., & Härter, M. (2008). Meta-analysis: Are 3 questions enough to detect unhealthy alcohol use? Annals of Internal Medicine, 149, 879–888.

    PubMed  Google Scholar 

  • Kuhnert, R., & Böhning, D. (2007). A comparison of three different models for estimating relative risk in meta-analysis. Statistics in Medicine, 28, 2277–2296.

    Article  Google Scholar 

  • Laird, N.M. (1978). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 73, 805–811.

    Article  Google Scholar 

  • Le, C.T. (2006). A solution for the most basic optimization problem associated with an ROC curve. Statistical Methods in Medical Research, 15, 571–584.

    Article  PubMed  Google Scholar 

  • Lindsay, B.G. (1983). The geometry of mixture likelihoods: A general theory. Annals of Statistics, 11, 86–94.

    Article  Google Scholar 

  • Lindsay, B.G. (1995). Mixture models: Theory, geometry, and applications. Hayward: Institute of Statistical Mathematics.

    Google Scholar 

  • Macaskill, P., Glasziou, P., & Irwig, L. (2005). Meta-analysis of diagnostic tests. In P. Armitage & T. Colton (Eds.), Encyclopedia of biostatistics. doi:10.1002/0470011815.b2a04026.

    Google Scholar 

  • Martin, C.S., Liepman, M.R., & Young, C.M. (1990). The Michigan alcoholism screening test: False positives in a college student sample. Alcoholism, Clinical and Experimental Research, 14, 853–855.

    Article  PubMed  Google Scholar 

  • McCullagh, P., & Nelder, J.A. (1989). Generalized linear models. London: Chapman & Hall.

    Google Scholar 

  • McLachlan, G., & Peel, D. (2000). Finite mixture models. New York: Wiley.

    Book  Google Scholar 

  • Midgette, A.S., Stukel, T.A., & Littenberg, B. (1993). A meta-analytic method for summarizing diagnostic test performances: Receiver-operating-characteristic-summary point estimates. Medical Decision Making, 13, 253–257.

    Article  PubMed  Google Scholar 

  • Mitchell, A.J. (2009). A meta-analysis of the accuracy of the mini-mental state examination in the detection of dementia and mild cognitive impairment. Journal of Psychiatric Research, 43, 411–431.

    Article  PubMed  Google Scholar 

  • Moses, L.E., Littenberg, B., & Shapiro, D. (1993). Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytical approaches and some additional considerations. Statistics in Medicine, 12, 1293–1316.

    PubMed  Google Scholar 

  • Ray, S., & Lindsay, B.G. (2008). Model selection in high dimensions: A quadratic-risk-based approach. Journal of the Royal Statistical Society. Series B, 70, 95–118.

    Google Scholar 

  • Rabe-Hesketh, S., Pickles, A., & Skrondal, A. (2003). Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation. Statistical Modelling, 3, 215–232.

    Article  Google Scholar 

  • Reinert, D.F., & Allen, J.P. (2002). The alcohol use identification test (AUDIT): A review of recent research. Alcoholism, Clinical and Experimental Research, 26, 272–279.

    Article  PubMed  Google Scholar 

  • Reitsma, J.B., Glas, A.S., Rutjes, A.W.S., Scholten, R.J.P.M., Bossuyt, P.M., & Zwinderman, A.H. (2005). Bivariate analysis of sensitivity and specificity produces informative measures in diagnostic reviews. Journal of Clinical Epidemiology, 58, 982–990.

    Article  PubMed  Google Scholar 

  • Ross, S. (1985). Introduction to probability models. Orlando: Academic Press.

    Google Scholar 

  • Rutter, C.M., & Gatsonis, C.A. (2001). A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine, 20, 2865–2884.

    Article  PubMed  Google Scholar 

  • Schulze, R., Holling, H., & Böhning, D. (Eds.) (2003). Meta-analysis. New developments and applications in medical and social sciences. Göttingen: Hogrefe & Huber.

    Google Scholar 

  • Selzer, M.L. (1971). The Michigan alcoholism screening test: The quest for a new diagnostic instrument. American Journal of Psychiatry, 127, 1653–1658.

    PubMed  Google Scholar 

  • Sidik, K., & Jonkman, J.N. (2005). Simple heterogeneity variance estimation for meta-analysis. Journal of the Royal Statistical Society. Series C, 54, 367–384.

    Article  Google Scholar 

  • Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Boca Raton: Chapman & Hall/CRC.

    Book  Google Scholar 

  • Storgaard, H., Nielsen, S.D., & Gluud, C. (1994). The validity of the Michigan alcoholism screening test (MAST). Alcohol and Alcoholism, 29, 493–502.

    PubMed  Google Scholar 

  • Sutton, A.J., Abrams, K.R., Jones, D.R., Sheldon, T.A., & Song, F. (2000). Methods for meta-analysis in medical research. New York: Wiley.

    Google Scholar 

  • Sweeting, M.J., Sutton, A.J., & Lambert, P.C. (2004). What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine, 23, 1351–1375.

    Article  PubMed  Google Scholar 

  • Swets, J.A. (2009). Signal detection theory and ROC analysis in psychology and diagnostics. New York: Psychology Press.

    Google Scholar 

  • van Houwelingen, H.C., Zwinderman, K.H., & Stijnen, T. (1993). A bivariate approach to meta-analysis. Statistics in Medicine, 12, 2273–2284.

    Article  PubMed  Google Scholar 

  • Whitehead, A. (2002). Meta-analysis of controlled clinical trials. New York: Wiley.

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Münster, Münster, Germany

    Heinz Holling & Walailuck Böhning

  2. School of Mathematics and Southampton Statistical Sciences Research Institute, University of Southampton, B39, R2021, Southampton, SO17 1BJ, UK

    Dankmar Böhning

Authors
  1. Heinz Holling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Walailuck Böhning
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dankmar Böhning
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dankmar Böhning.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Holling, H., Böhning, W. & Böhning, D. Likelihood-Based Clustering of Meta-Analytic SROC Curves. Psychometrika 77, 106–126 (2012). https://doi.org/10.1007/s11336-011-9236-2

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-011-9236-2

Key words

Search

Navigation