, Volume 80, Issue 4, pp 1084–1104 | Cite as

Meta-analysis of Diagnostic Accuracy and ROC Curves with Covariate Adjusted Semiparametric Mixtures

  • Philipp DoeblerEmail author
  • Heinz Holling


Many screening tests dichotomize a measurement to classify subjects. Typically a cut-off value is chosen in a way that allows identification of an acceptable number of cases relative to a reference procedure, but does not produce too many false positives at the same time. Thus for the same sample many pairs of sensitivities and false positive rates result as the cut-off is varied. The curve of these points is called the receiver operating characteristic (ROC) curve. One goal of diagnostic meta-analysis is to integrate ROC curves and arrive at a summary ROC (SROC) curve. Holling, Böhning, and Böhning (Psychometrika 77:106–126, 2012a) demonstrated that finite semiparametric mixtures can describe the heterogeneity in a sample of Lehmann ROC curves well; this approach leads to clusters of SROC curves of a particular shape. We extend this work with the help of the \(t_{\alpha }\) transformation, a flexible family of transformations for proportions. A collection of SROC curves is constructed that approximately contains the Lehmann family but in addition allows the modeling of shapes beyond the Lehmann ROC curves. We introduce two rationales for determining the shape from the data. Using the fact that each curve corresponds to a natural univariate measure of diagnostic accuracy, we show how covariate adjusted mixtures lead to a meta-regression on SROC curves. Three worked examples illustrate the method.


diagnostic meta-analysis summary receiver operating characteristic curve transformations semiparametric mixtures meta-regression 



This work was funded by DFG Grant HO 1286/7-2.


  1. Aertgeerts, B., Buntinx, F., Ansoms, S., & Fevery, J. (2001). Screening properties of questionnaires and laboratory tests for the detection of alcohol abuse or dependence in a general practice population. The British Journal of General Practice, 51, 206–217.PubMedCentralPubMedGoogle Scholar
  2. Aitkin, M. (1999a). A general maximum likelihood analysis of variance components in generalized linear models. Biometrics, 55, 117–128.CrossRefPubMedGoogle Scholar
  3. Aitkin, M. (1999b). Meta-analysis by random effect modelling in generalized linear models. Statistics in Medicine, 18, 2343–2351.CrossRefPubMedGoogle Scholar
  4. Aranda-Ordaz, F. (1981). On two families of transformations to additivity for binary response data. Biometrika, 68, 357–363.CrossRefGoogle Scholar
  5. Arends, L., Hamza, T., Van Houwelingen, J., Heijenbrok-Kal, M., Hunink, M., & Stijnen, T. (2008). Bivariate random effects meta-analysis of ROC curves. Medical Decision Making, 28, 621–638.CrossRefPubMedGoogle Scholar
  6. Böhning, D. (2000). Computer-assisted analysis of mixtures and applications: Meta-analysis, disease mapping and others., Monographs on statistics and applied probability 81 Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
  7. Böhning, D., Dietz, E., & Schlattmann, P. (1998). Recent developments in computer-assisted analysis of mixtures. Biometrics, 54, 525–536.CrossRefPubMedGoogle Scholar
  8. Böhning, D., Böhning, W., & Holling, H. (2008). Revisiting Youden’s index as a useful measure of the misclassification error in meta-analysis of diagnostic studies. Statistical Methods in Medical Research, 17, 543–554.CrossRefPubMedGoogle Scholar
  9. Brent, R. P. (1973). Algorithms for minimization without derivatives. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  10. Bush, K., Kivlahan, D., McDonell, M., Fihn, S., & Bradley, K. (1998). The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Archives of Internal Medicine, 158, 1789–1795.CrossRefPubMedGoogle Scholar
  11. Chu, H., & Cole, S. R. (2006). Bivariate meta-analysis of sensitivity and specificity with sparse data: A generalized linear mixed model approach. Journal of Clinical Epidemiology, 59(12), 1331–1332.CrossRefPubMedGoogle Scholar
  12. Deeks, J. (2001). Systematic reviews of evaluations of diagnostic and screening tests. British Medical Journal, 323, 157–162.PubMedCentralCrossRefPubMedGoogle Scholar
  13. DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.CrossRefPubMedGoogle Scholar
  14. Doebler, P., Holling, H., & Böhning, D. (2012). A mixed model approach to meta-analysis of diagnostic studies with binary test outcome. Psychological Methods, 17, 418–436.CrossRefPubMedGoogle Scholar
  15. Efron, B. (1987). Better bootstrap confidence intervals. Journal of the American Statistical Association, 82(397), 171–185.CrossRefGoogle Scholar
  16. Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189–198.CrossRefPubMedGoogle Scholar
  17. Glas, A., Lijmer, J., Prins, M., Bonsel, G., & Bossuyt, P. (2003). The diagnostic odds ratio: A single indicator of test performance. Journal of Clinical Epidemiology, 56, 1129–1135.CrossRefPubMedGoogle Scholar
  18. Gordon, A., Maisto, S., McNeil, M., Kraemer, K., Conigliaro, R., Kelley, M., et al. (2001). Three questions can detect hazardous drinkers. Journal of Family Practice, 50, 313–320.PubMedGoogle Scholar
  19. Guerrero, V., & Johnson, R. (1982). Use of the Box–Cox transformation with binary response models. Biometrika, 69, 309–314.CrossRefGoogle Scholar
  20. Hamza, T., Reitsma, J., & Stijnen, T. (2008). Meta-analysis of diagnostic studies: A comparison of random intercept, normal–normal, and binomial–normal bivariate summary ROC approaches. Medical Decision Making, 28, 639–649.CrossRefPubMedGoogle Scholar
  21. Harbord, R., Deeks, J., Egger, M., Whiting, P., & Sterne, J. (2007). A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics, 8, 239–251.CrossRefPubMedGoogle Scholar
  22. Holling, H., Böhning, W., & Böhning, D. (2012a). Likelihood-based clustering of meta-analytic SROC curves. Psychometrika, 77, 106–126.CrossRefGoogle Scholar
  23. Holling, H., Böhning, W., & Böhning, D. (2012b). Meta-analysis of diagnostic studies based upon SROC-curves: A mixed model approach using the Lehmann family. Statistical Modelling, 12, 347–375.CrossRefGoogle Scholar
  24. Keribin, C. (2000). Consistent estimation of the order of mixture models. Sankhyā: The Indian Journal of Statistics, Series A, 62, 49–66.Google Scholar
  25. Koenker, R., & Yoon, J. (2009). Parametric links for binary choice models: A Fisherian–Bayesian colloquy. Journal of Econometrics, 152, 120–130.CrossRefGoogle Scholar
  26. Kriston, L., Hölzel, L., Weiser, A., Berner, M., & Härter, M. (2008). Meta-analysis: Are 3 questions enough to detect unhealthy alcohol use? Annals of Internal Medicine, 149, 879–888.CrossRefPubMedGoogle Scholar
  27. Le, C. (2006). A solution for the most basic optimization problem associated with an ROC curve. Statistical Methods in Medical Research, 15, 571–584.CrossRefPubMedGoogle Scholar
  28. Lindsay, B. (1983). The geometry of mixture likelihoods: A general theory. The Annals of Statistics, 11, 86–94.CrossRefGoogle Scholar
  29. Lindsay, B. (1995). Mixture models: Theory, geometry and applications. NSF-CBMS regional conference series in probability and statistics.Google Scholar
  30. Littenberg, B., & Moses, L. (1993). Estimating diagnostic accuracy from multiple conflicting reports: A new meta-analytic method. Medical Decision Making, 13, 313–321.CrossRefPubMedGoogle Scholar
  31. Macaskill, P., Gatsonis, C., Deeks, J., Harbord, R., & Takwoingi, Y. (2010). Chapter 10: Analysing and presenting results. In J. Deeks, P. Bossuyt & C. Gatsonis (Eds.), Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0. The Cochrane Collaboration. Retrieved from: Accessed 25 Oct 2014.
  32. Ma, X., Nie, L., Cole, S. R., & Chu, H. (2013). Statistical methods for multivariate meta-analysis of diagnostic tests: An overview and tutorial. Statistical Methods in Medical Research. doi: 10.1177/0962280213492588.
  33. McCullagh, P., & Nelder, J. (1989). Generalized linear models. Boca Raton, FL: Chapman & Hall/CRC.CrossRefGoogle Scholar
  34. Mitchell, A. (2009). A meta-analysis of the accuracy of the mini-mental state examination in the detection of dementia and mild cognitive impairment. Journal of Psychiatric Research, 43, 411–431.CrossRefPubMedGoogle Scholar
  35. Moses, L., Shapiro, D., & Littenberg, B. (1993). Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations. Statistics in Medicine, 12, 1293–1316.CrossRefPubMedGoogle Scholar
  36. Patrick, D., Cheadle, A., Thompson, D., Diehr, P., Koepsell, T., & Kinne, S. (1994). The validity of self-reported smoking: A review and meta-analysis. American Journal of Public Health, 84, 1086–1093.PubMedCentralCrossRefPubMedGoogle Scholar
  37. Pepe, M. (2000). Receiver operating characteristic methodology. Journal of the American Statistical Association, 95, 308–311.CrossRefGoogle Scholar
  38. Pepe, M. (2004). The statistical evaluation of medical tests for classification and prediction. Oxford, UK: Oxford University Press.Google Scholar
  39. Piepho, H. (2003). The folded exponential transformation for proportions. Journal of the Royal Statistical Society: Series D, 52, 575–589.Google Scholar
  40. Prentice, R. (1976). A generalization of the probit and logit methods for dose response curves. Biometrics, 32(4), 761–768.Google Scholar
  41. Reitsma, J., Glas, A., Rutjes, A., Scholten, R., Bossuyt, P., & Zwinderman, A. (2005). Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology, 58, 982–990.CrossRefPubMedGoogle Scholar
  42. Rücker, G., & Schumacher, M. (2010). Summary ROC curve based on a weighted Youden index for selecting an optimal cutpoint in meta-analysis of diagnostic accuracy. Statistics in Medicine, 29, 3069–3078.CrossRefPubMedGoogle Scholar
  43. Rumpf, H., Hapke, U., Meyer, C., & John, U. (2002). Screening for alcohol use disorders and at-risk drinking in the general population: Psychometric performance of three questionnaires. Alcohol and Alcoholism, 37, 261–268.CrossRefPubMedGoogle Scholar
  44. Rutter, C., & Gatsonis, C. (2001). A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine, 20, 2865–2884.CrossRefPubMedGoogle Scholar
  45. Schlattmann, P. (2009). Medical applications of finite mixture models. New York, NY: Springer.Google Scholar
  46. Schlattmann, P. & Hoehne, J. (2013). CAMAN: Finite mixture models and meta-analysis tools—based on C.A.MAN. R package version 0.67.Google Scholar
  47. Sweeting, M., Sutton, A., & Lambert, P. (2004). What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine, 23, 1351–1375.CrossRefPubMedGoogle Scholar
  48. Walter, S. (2002). Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data. Statistics in Medicine, 21(9), 1237–1256.CrossRefPubMedGoogle Scholar
  49. Wedel, M. (2002). Concomitant variables in finite mixture models. Statistica Neerlandica, 56, 362–375.CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2014

Authors and Affiliations

  1. 1.Institut für PsychologieWestfälische Wilhelms-UniversitätMünsterGermany

Personalised recommendations