Skip to main content
SpringerLink
Log in

Meta-analysis of Diagnostic Accuracy and ROC Curves with Covariate Adjusted Semiparametric Mixtures

Psychometrika Aims and scope Submit manuscript

Cite this article

Abstract

Many screening tests dichotomize a measurement to classify subjects. Typically a cut-off value is chosen in a way that allows identification of an acceptable number of cases relative to a reference procedure, but does not produce too many false positives at the same time. Thus for the same sample many pairs of sensitivities and false positive rates result as the cut-off is varied. The curve of these points is called the receiver operating characteristic (ROC) curve. One goal of diagnostic meta-analysis is to integrate ROC curves and arrive at a summary ROC (SROC) curve. Holling, Böhning, and Böhning (Psychometrika 77:106–126, 2012a) demonstrated that finite semiparametric mixtures can describe the heterogeneity in a sample of Lehmann ROC curves well; this approach leads to clusters of SROC curves of a particular shape. We extend this work with the help of the \(t_{\alpha }\) transformation, a flexible family of transformations for proportions. A collection of SROC curves is constructed that approximately contains the Lehmann family but in addition allows the modeling of shapes beyond the Lehmann ROC curves. We introduce two rationales for determining the shape from the data. Using the fact that each curve corresponds to a natural univariate measure of diagnostic accuracy, we show how covariate adjusted mixtures lead to a meta-regression on SROC curves. Three worked examples illustrate the method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. In this paragraph sensitivity does not refer to the true positive rate.

  2. To answer the question by how much the same questionnaire improves in accuracy if we administer it by interviewers instead of self-administration, the data of Patrick et al. (1994) is not directly suited, since it does not focus on primary studies performing such a comparison.

References

  • Aertgeerts, B., Buntinx, F., Ansoms, S., & Fevery, J. (2001). Screening properties of questionnaires and laboratory tests for the detection of alcohol abuse or dependence in a general practice population. The British Journal of General Practice, 51, 206–217.

    PubMed Central  PubMed  Google Scholar 

  • Aitkin, M. (1999a). A general maximum likelihood analysis of variance components in generalized linear models. Biometrics, 55, 117–128.

    Article  PubMed  Google Scholar 

  • Aitkin, M. (1999b). Meta-analysis by random effect modelling in generalized linear models. Statistics in Medicine, 18, 2343–2351.

    Article  PubMed  Google Scholar 

  • Aranda-Ordaz, F. (1981). On two families of transformations to additivity for binary response data. Biometrika, 68, 357–363.

    Article  Google Scholar 

  • Arends, L., Hamza, T., Van Houwelingen, J., Heijenbrok-Kal, M., Hunink, M., & Stijnen, T. (2008). Bivariate random effects meta-analysis of ROC curves. Medical Decision Making, 28, 621–638.

    Article  PubMed  Google Scholar 

  • Böhning, D. (2000). Computer-assisted analysis of mixtures and applications: Meta-analysis, disease mapping and others., Monographs on statistics and applied probability 81 Boca Raton, FL: Chapman & Hall/CRC.

    Google Scholar 

  • Böhning, D., Dietz, E., & Schlattmann, P. (1998). Recent developments in computer-assisted analysis of mixtures. Biometrics, 54, 525–536.

    Article  PubMed  Google Scholar 

  • Böhning, D., Böhning, W., & Holling, H. (2008). Revisiting Youden’s index as a useful measure of the misclassification error in meta-analysis of diagnostic studies. Statistical Methods in Medical Research, 17, 543–554.

    Article  PubMed  Google Scholar 

  • Brent, R. P. (1973). Algorithms for minimization without derivatives. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  • Bush, K., Kivlahan, D., McDonell, M., Fihn, S., & Bradley, K. (1998). The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Archives of Internal Medicine, 158, 1789–1795.

    Article  PubMed  Google Scholar 

  • Chu, H., & Cole, S. R. (2006). Bivariate meta-analysis of sensitivity and specificity with sparse data: A generalized linear mixed model approach. Journal of Clinical Epidemiology, 59(12), 1331–1332.

    Article  PubMed  Google Scholar 

  • Deeks, J. (2001). Systematic reviews of evaluations of diagnostic and screening tests. British Medical Journal, 323, 157–162.

    Article  PubMed Central  PubMed  Google Scholar 

  • DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.

    Article  PubMed  Google Scholar 

  • Doebler, P., Holling, H., & Böhning, D. (2012). A mixed model approach to meta-analysis of diagnostic studies with binary test outcome. Psychological Methods, 17, 418–436.

    Article  PubMed  Google Scholar 

  • Efron, B. (1987). Better bootstrap confidence intervals. Journal of the American Statistical Association, 82(397), 171–185.

    Article  Google Scholar 

  • Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189–198.

    Article  PubMed  Google Scholar 

  • Glas, A., Lijmer, J., Prins, M., Bonsel, G., & Bossuyt, P. (2003). The diagnostic odds ratio: A single indicator of test performance. Journal of Clinical Epidemiology, 56, 1129–1135.

    Article  PubMed  Google Scholar 

  • Gordon, A., Maisto, S., McNeil, M., Kraemer, K., Conigliaro, R., Kelley, M., et al. (2001). Three questions can detect hazardous drinkers. Journal of Family Practice, 50, 313–320.

    PubMed  Google Scholar 

  • Guerrero, V., & Johnson, R. (1982). Use of the Box–Cox transformation with binary response models. Biometrika, 69, 309–314.

    Article  Google Scholar 

  • Hamza, T., Reitsma, J., & Stijnen, T. (2008). Meta-analysis of diagnostic studies: A comparison of random intercept, normal–normal, and binomial–normal bivariate summary ROC approaches. Medical Decision Making, 28, 639–649.

    Article  PubMed  Google Scholar 

  • Harbord, R., Deeks, J., Egger, M., Whiting, P., & Sterne, J. (2007). A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics, 8, 239–251.

    Article  PubMed  Google Scholar 

  • Holling, H., Böhning, W., & Böhning, D. (2012a). Likelihood-based clustering of meta-analytic SROC curves. Psychometrika, 77, 106–126.

    Article  Google Scholar 

  • Holling, H., Böhning, W., & Böhning, D. (2012b). Meta-analysis of diagnostic studies based upon SROC-curves: A mixed model approach using the Lehmann family. Statistical Modelling, 12, 347–375.

    Article  Google Scholar 

  • Keribin, C. (2000). Consistent estimation of the order of mixture models. Sankhyā: The Indian Journal of Statistics, Series A, 62, 49–66.

    Google Scholar 

  • Koenker, R., & Yoon, J. (2009). Parametric links for binary choice models: A Fisherian–Bayesian colloquy. Journal of Econometrics, 152, 120–130.

    Article  Google Scholar 

  • Kriston, L., Hölzel, L., Weiser, A., Berner, M., & Härter, M. (2008). Meta-analysis: Are 3 questions enough to detect unhealthy alcohol use? Annals of Internal Medicine, 149, 879–888.

    Article  PubMed  Google Scholar 

  • Le, C. (2006). A solution for the most basic optimization problem associated with an ROC curve. Statistical Methods in Medical Research, 15, 571–584.

    Article  PubMed  Google Scholar 

  • Lindsay, B. (1983). The geometry of mixture likelihoods: A general theory. The Annals of Statistics, 11, 86–94.

    Article  Google Scholar 

  • Lindsay, B. (1995). Mixture models: Theory, geometry and applications. NSF-CBMS regional conference series in probability and statistics.

  • Littenberg, B., & Moses, L. (1993). Estimating diagnostic accuracy from multiple conflicting reports: A new meta-analytic method. Medical Decision Making, 13, 313–321.

    Article  PubMed  Google Scholar 

  • Macaskill, P., Gatsonis, C., Deeks, J., Harbord, R., & Takwoingi, Y. (2010). Chapter 10: Analysing and presenting results. In J. Deeks, P. Bossuyt & C. Gatsonis (Eds.), Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0. The Cochrane Collaboration. Retrieved from: http://srdta.cochrane.org/. Accessed 25 Oct 2014.

  • Ma, X., Nie, L., Cole, S. R., & Chu, H. (2013). Statistical methods for multivariate meta-analysis of diagnostic tests: An overview and tutorial. Statistical Methods in Medical Research. doi:10.1177/0962280213492588.

  • McCullagh, P., & Nelder, J. (1989). Generalized linear models. Boca Raton, FL: Chapman & Hall/CRC.

    Book  Google Scholar 

  • Mitchell, A. (2009). A meta-analysis of the accuracy of the mini-mental state examination in the detection of dementia and mild cognitive impairment. Journal of Psychiatric Research, 43, 411–431.

    Article  PubMed  Google Scholar 

  • Moses, L., Shapiro, D., & Littenberg, B. (1993). Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations. Statistics in Medicine, 12, 1293–1316.

    Article  PubMed  Google Scholar 

  • Patrick, D., Cheadle, A., Thompson, D., Diehr, P., Koepsell, T., & Kinne, S. (1994). The validity of self-reported smoking: A review and meta-analysis. American Journal of Public Health, 84, 1086–1093.

    Article  PubMed Central  PubMed  Google Scholar 

  • Pepe, M. (2000). Receiver operating characteristic methodology. Journal of the American Statistical Association, 95, 308–311.

    Article  Google Scholar 

  • Pepe, M. (2004). The statistical evaluation of medical tests for classification and prediction. Oxford, UK: Oxford University Press.

    Google Scholar 

  • Piepho, H. (2003). The folded exponential transformation for proportions. Journal of the Royal Statistical Society: Series D, 52, 575–589.

    Google Scholar 

  • Prentice, R. (1976). A generalization of the probit and logit methods for dose response curves. Biometrics, 32(4), 761–768.

  • Reitsma, J., Glas, A., Rutjes, A., Scholten, R., Bossuyt, P., & Zwinderman, A. (2005). Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology, 58, 982–990.

    Article  PubMed  Google Scholar 

  • Rücker, G., & Schumacher, M. (2010). Summary ROC curve based on a weighted Youden index for selecting an optimal cutpoint in meta-analysis of diagnostic accuracy. Statistics in Medicine, 29, 3069–3078.

    Article  PubMed  Google Scholar 

  • Rumpf, H., Hapke, U., Meyer, C., & John, U. (2002). Screening for alcohol use disorders and at-risk drinking in the general population: Psychometric performance of three questionnaires. Alcohol and Alcoholism, 37, 261–268.

    Article  PubMed  Google Scholar 

  • Rutter, C., & Gatsonis, C. (2001). A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine, 20, 2865–2884.

    Article  PubMed  Google Scholar 

  • Schlattmann, P. (2009). Medical applications of finite mixture models. New York, NY: Springer.

    Google Scholar 

  • Schlattmann, P. & Hoehne, J. (2013). CAMAN: Finite mixture models and meta-analysis tools—based on C.A.MAN. R package version 0.67.

  • Sweeting, M., Sutton, A., & Lambert, P. (2004). What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine, 23, 1351–1375.

    Article  PubMed  Google Scholar 

  • Walter, S. (2002). Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data. Statistics in Medicine, 21(9), 1237–1256.

    Article  PubMed  Google Scholar 

  • Wedel, M. (2002). Concomitant variables in finite mixture models. Statistica Neerlandica, 56, 362–375.

    Article  Google Scholar 

Download references

Acknowledgments

This work was funded by DFG Grant HO 1286/7-2.

Author information

Authors and Affiliations

  1. Institut für Psychologie, Westfälische Wilhelms-Universität, Fliednerstr. 21, 48149 , Münster, Germany

    Philipp Doebler & Heinz Holling

Authors
  1. Philipp Doebler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heinz Holling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Doebler.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Doebler, P., Holling, H. Meta-analysis of Diagnostic Accuracy and ROC Curves with Covariate Adjusted Semiparametric Mixtures. Psychometrika 80, 1084–1104 (2015). https://doi.org/10.1007/s11336-014-9430-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-014-9430-0

Keywords

search

Navigation