Appraising Heterogeneity

  • Antonia ZapfEmail author


In general, in the meta-analysis of diagnostic accuracy studies, large heterogeneity between the studies is present, which cannot be explained by chance alone and can lead to inconsistent results. There are several study-specific aspects, which can lead to large variation or even to bias. Therefore, methods are needed to evaluate and to consider this heterogeneity in an appropriate way. For the analysis the most noted are the HSROC approach and the bivariate model. However, in the literature several other approaches can be found, which consider heterogeneity in specific scenarios. In this chapter the different sources of heterogeneity will be described, and methods for illustration and quantification of the heterogeneity will be presented. Furthermore, the available approaches for meta-analysis with covariates will be discussed with the focus on sensitivity (as true positive rate) and specificity (as true negative rate) as co-primary endpoints and the comparison of an index test with the reference standard. Furthermore, some extra topics will be addressed, as, for example, meta-analysis of diagnostic studies with multiple thresholds, individual data meta-analysis, and other endpoints like predictive values. Data from a published meta-analysis will be used for illustration.


Heterogeneity Meta-regression HSROC Bivariate model Diagnostic accuracy 


  1. 1.
    Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0: The Cochrane Collaboration; 2010. Accessed 29 June 2018.
  2. 2.
    Naaktgeboren CA, van Enst WA, Ochodo EA, de Groot JA, Hooft L, Leeflang MM, et al. Systematic overview finds variation in approaches to investigating and reporting on sources of heterogeneity in systematic reviews of diagnostic studies. J Clin Epidemiol. 2014;67:1200–9.PubMedCrossRefGoogle Scholar
  3. 3.
    Naaktgeboren CA, Ochodo EA, Van Enst WA, de Groot JAH, Hooft L, Leeflang MMG, et al. Assessing variability in results in systematic reviews of diagnostic studies. BMC Med Res Methodol. 2016;16:6.PubMedPubMedCentralCrossRefGoogle Scholar
  4. 4.
    Roberts E, Ludman AJ, Dworzynski K, Al-Mohammad A, Cowie MR, McMurray JJ, et al. The diagnostic accuracy of the natriuretic peptides in heart failure: systematic review and diagnostic meta-analysis in the acute care setting. BMJ. 2015;350:h910.PubMedPubMedCentralCrossRefGoogle Scholar
  5. 5.
    Doebler P, Holling H. Meta-analysis of diagnostic accuracy with mada. 2015. 29 June 2018.Google Scholar
  6. 6.
    Doebler P. Mada: meta-analysis of diagnostic accuracy. R package version 0.5.7. 2015. Accessed 29 June 2018.
  7. 7.
    Schwarzer G. Package ‘meta’. 2017. Accessed 29 June 2018.
  8. 8.
    Schwarzer G. meta: An R package for meta-analysis. R News. 2007;7:40–5.Google Scholar
  9. 9.
    R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2017. Accessed 29 June 2018.
  10. 10.
    Alibay Y, Beauchet A, El Mahmoud R, Schmitta C, Brun-Neyd D, Benoite M, et al. Plasma N-terminal pro-brain natriuretic peptide and brain natriuretic peptide in assessment of acute dyspnea. Biomed Pharmacother. 2005;59:20–4.PubMedCrossRefGoogle Scholar
  11. 11.
    Arques S, Roux E, Sbragia P, Gelisse R, Ambrosi P, Pieri B, et al. Comparative accuracy of color M-mode and tissue Doppler echocardiography in the emergency diagnosis of congestive heart failure in chronic hypertensive patients with normal left ventricular ejection fraction. Am J Cardiol. 2005;96:1456–9.PubMedCrossRefGoogle Scholar
  12. 12.
    Arques S, Roux E, Sbragia P, Pieri B, Gelisse R, Luccioni R, et al. Usefulness of bedside tissue Doppler echocardiography and B-type natriuretic peptide (BNP) in differentiating congestive heart failure from noncardiac cause of acute dyspnea in elderly patients with a normal left ventricular ejection fraction and permanent, nonvalvular atrial fibrillation: insights from a prospective, monocenter study. Echocardiography. 2007;24:499–507.PubMedCrossRefGoogle Scholar
  13. 13.
    Barcarse E, Kazanegra R, Chen A, Chiu A, Clopton P, Maisel A. Combination of B-type natriuretic peptide levels and non-invasive hemodynamic parameters in diagnosing congestive heart failure in the emergency department. Congest Heart Fail. 2004;10:171–6.PubMedCrossRefGoogle Scholar
  14. 14.
    Blonde-Cynober F, Morineau G, Estrugo B, Fillie E, Aussel C, Vincent J-P. Diagnostic and prognostic value of brain natriuretic peptide (BNP) concentrations in very elderly heart disease patients: specific geriatric cut-off and impacts of age, gender, renal dysfunction, and nutritional status. Arch Gerontol Geriatr. 2011;52:106–10.PubMedCrossRefGoogle Scholar
  15. 15.
    Chenevier-Gobeaux C, Guerin S, Andre S, Ray P, Cynober L, Gestin S, et al. Midregional pro-atrial natriuretic peptide for the diagnosis of cardiac-related dyspnea according to renal function in the emergency department: a comparison with B-type natriuretic peptide (BNP) and N-terminal proBNP. Clin Chem. 2010;56:1708–17.PubMedCrossRefGoogle Scholar
  16. 16.
    Chung T, Sindone A, Foo F, Dwyer A, Paoloni R, Janu MR, et al. Influence of history of heart failure on diagnostic performance and utility of B-type natriuretic peptide testing for acute dyspnea in the emergency department. Am Heart J. 2006;152:949–55.PubMedCrossRefGoogle Scholar
  17. 17.
    Dao Q, Krishnaswamy P, Kazanegra R, Harrison A, Amirnovin R, Lenert L, et al. Utility of B-type natriuretic peptide in the diagnosis of congestive heart failure in an urgent-care setting. J Am Coll Cardiol. 2001;37:379–85.PubMedCrossRefGoogle Scholar
  18. 18.
    Davis M, Espiner E, Richards G, Billings J, Town I, Neill A, et al. Plasma brain natriuretic peptide in assessment of acute dyspnoea. Lancet. 1994;343:440–4.PubMedCrossRefGoogle Scholar
  19. 19.
    Dokainish H, Zoghbi WA, Lakkis NM, Quinones MA, Nagueh SF. Comparative accuracy of B-type natriuretic peptide and tissue Doppler echocardiography in the diagnosis of congestive heart failure. Am J Cardiol. 2004;93:1130–5.PubMedCrossRefGoogle Scholar
  20. 20.
    Fleischer D, Espiner EA, Yandle TG, Livesey JH, Billings J, Town I, et al. Rapid assay of plasma brain natriuretic peptide in the assessment of acute dyspnoea. N Z Med J. 1997;110:71–4.PubMedGoogle Scholar
  21. 21.
    Gorissen C, Baumgarten R, de Groot M, van Haren E, Kragten H, Leers M. Analytical and clinical performance of three natriuretic peptide tests in the emergency room. Clin Chem Lab Med. 2007;45:678–84.PubMedCrossRefGoogle Scholar
  22. 22.
    Karmpaliotis D, Kirtane AJ, Ruisi CP, Polonsky T, Malhotra A, Talmor D, et al. Diagnostic and prognostic utility of brain natriuretic Peptide in subjects admitted to the ICU with hypoxic respiratory failure due to noncardiogenic and cardiogenic pulmonary edema. Chest. 2007;131:964–71.PubMedPubMedCentralCrossRefGoogle Scholar
  23. 23.
    Lainchbury JG, Campbell E, Frampton CM, Yandle TG, Nicholls MG, Richards AM. Brain natriuretic peptide and n-terminal brain natriuretic peptide in the diagnosis of heart failure in patients with acute shortness of breath. J Am Coll Cardiol. 2003;42:728–35.PubMedCrossRefGoogle Scholar
  24. 24.
    Logeart D, Saudubray C, Beyne P, Thabut G, Ennezat PV, Chavelas C, et al. Comparative value of Doppler echocardiography and B-type natriuretic peptide assay in the etiologic diagnosis of acute dyspnea. J Am Coll Cardiol. 2002;40:1794–800.PubMedCrossRefGoogle Scholar
  25. 25.
    Lokuge A, Lam L, Cameron P, Krum H, de Villiers S, Bystrzycki A, et al. B-type natriuretic peptide testing and the accuracy of heart failure diagnosis in the emergency department. Circ Heart Fail. 2010;3:104–10.PubMedCrossRefGoogle Scholar
  26. 26.
    Maisel AS, Krishnaswamy P, Nowak RM, McCord J, Hollander JE, Duc P, et al. Rapid measurement of B-type natriuretic peptide in the emergency diagnosis of heart failure. N Engl J Med. 2002;347:161–7.PubMedCrossRefPubMedCentralGoogle Scholar
  27. 27.
    Maisel A, Mueller C, Nowak R, Peacock WF, Landsberg JW, Ponikowski P, et al. Mid-region pro-hormone markers for diagnosis and prognosis in acute dyspnea: results from the BACH (Biomarkers in Acute Heart Failure) trial. J Am Coll Cardiol. 2010;55:2062–76.PubMedCrossRefGoogle Scholar
  28. 28.
    Mueller T, Gegenhuber A, Poelz W, Haltmayer M. Diagnostic accuracy of B type natriuretic peptide and amino terminal proBNP in the emergency diagnosis of heart failure. Heart. 2005;91:606–12.PubMedPubMedCentralCrossRefGoogle Scholar
  29. 29.
    Parab R, Vasudevan A, Brensilver J, Gitler B. Utility of brain natriuretic peptide as a diagnostic tool for congestive heart failure in the elderly. Crit Pathw Cardiol. 2005;4:140–4.PubMedCrossRefGoogle Scholar
  30. 30.
    Ray P, Arthaud M, Lefort Y, Birolleau S, Beigelman C, Riou B, et al. Usefulness of B-type natriuretic peptide in elderly patients with acute dyspnea. Intensive Care Med. 2004;30:2230–6.PubMedCrossRefGoogle Scholar
  31. 31.
    Kevin Rogers R, Stehlik J, Stoddard GJ, Greene T, Collins SP, Peacock WF, et al. Adjusting for clinical covariates improves the ability of B-type natriuretic peptide to distinguish cardiac from non-cardiac dyspnoea: a sub-study of HEARD-IT. Eur J Heart Fail. 2009;11:1043–9.PubMedCrossRefGoogle Scholar
  32. 32.
    Sanz MP, Borque L, Rus A, Vicente B, Ramirez Y, Lasa L. Comparison of BNP and NT-proBNP assays in the approach to the emergency diagnosis of acute dyspnea. J Clin Lab Anal. 2006;20:227–32.PubMedCrossRefGoogle Scholar
  33. 33.
    Villacorta H, Duarte A, Duarte NM, Carrano A, Mesquita ET, Dohmann HJ, et al. The role of B-type natriuretic peptide in the diagnosis of congestive heart failure in patients presenting to an emergency department with dyspnea. Arq Bras Cardiol. 2002;79:569–72.PubMedCrossRefGoogle Scholar
  34. 34.
    Wang HK, Tsai MS, Chang JH, Wang TD, Chen WJ, Huang CH. Cardiac ultrasound helps for differentiating the causes of acute dyspnea with available B-type natriuretic peptide tests. Am J Emerg Med. 2010;28:987–93.PubMedCrossRefGoogle Scholar
  35. 35.
    Lijmer JG, Bossuyt PM, Heisterkamp SH. Exploring sources of heterogeneity in systematic reviews of diagnostic tests. Stat Med. 2002;21:1525–37.PubMedCrossRefGoogle Scholar
  36. 36.
    Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PMM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy. Ann Intern Med. 2004;140:189–202.PubMedCrossRefGoogle Scholar
  37. 37.
    Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529–36.CrossRefGoogle Scholar
  38. 38.
    Whiting P, Rutjes AWS, Reitsma JB, Bossuyt PMM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25.PubMedPubMedCentralCrossRefGoogle Scholar
  39. 39.
    Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174:469–76.PubMedPubMedCentralCrossRefGoogle Scholar
  40. 40.
    Song JW, Kim HM, Bellfi LT, Chung KC. The effect of study design biases on the diagnostic accuracy of magnetic resonance imaging for detecting silicone breast implant ruptures: a meta-analysis. Plast Reconstr Surg. 2011;127:1029–44.PubMedPubMedCentralCrossRefGoogle Scholar
  41. 41.
    Parker LA, Saez NG, Porta M, Hernández-Aguado I, Lumbreras B. The impact of including different study designs in meta-analyses of diagnostic accuracy studies. Eur J Epidemiol. 2013;28:713–20.PubMedCrossRefGoogle Scholar
  42. 42.
    Leeflang MM, Rutjes AW, Reitsma JB, Hooft L, Bossuyt PM. Variation of a test’s sensitivity and specificity with disease prevalence. CMAJ. 2013;185:E537–44.PubMedPubMedCentralCrossRefGoogle Scholar
  43. 43.
    Ochodo EA, van Enst WA, Naaktgeboren CA, de Groot JAH, Hooft L, Moons KGM, et al. Incorporating quality assessments of primary studies in the conclusions of diagnostic accuracy reviews: a cross-sectional study. BMC Med Res Methodol. 2014;14:33.PubMedPubMedCentralCrossRefGoogle Scholar
  44. 44.
    Cohen JF, Korevaar DA, Wang J, Leeflang MM, Bossuyt PM. Meta-epidemiologic study showed frequent time trends in summary estimates from meta-analyses of diagnostic accuracy studies. J Clin Epidemiol. 2016;77:60–7.PubMedCrossRefGoogle Scholar
  45. 45.
    Moses LE, Shapiro DE, Littenberg B. Combining independent studies of a diagnostic tests into a summary ROC curve: Data-analytic approaches and some additional considerations. Stat Med. 1993;12:1293–316.CrossRefPubMedGoogle Scholar
  46. 46.
    Leeflang MMG, Deeks JJ, Rutjes AWS, Reitsma JB, Bossuyt PMM. Bivariate meta-analysis of predictive values of diagnostic tests can be an alternative to bivariate meta-analysis of sensitivity and specificity. J Clin Epidemiol. 2012;65:1088–97.PubMedCrossRefGoogle Scholar
  47. 47.
    Phillips B, Stewart LA, Sutton AJ. ‘Cross hairs’ plots for diagnostic meta-analysis. Res Synth Methods. 2010;1:308–15.PubMedPubMedCentralCrossRefGoogle Scholar
  48. 48.
    Light RJ, Pillemer DB. Summing up: the science of reviewing research. Cambridge, Massachusetts: Harvard University Press; 1984.Google Scholar
  49. 49.
    Song F, Khan KS, Dinnes J, Sutton AJ. Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy. Int J Epidemiol. 2002;31:88–95.PubMedCrossRefGoogle Scholar
  50. 50.
    Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005;58:882–93.CrossRefPubMedGoogle Scholar
  51. 51.
    Glas AS, Lijmer JG, Prins MH, Bonsel GJ, Bossuyt PM. The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol. 2003;56:1129–35.PubMedPubMedCentralCrossRefGoogle Scholar
  52. 52.
    Jackson D, White IR, Thompson SG. Extending DerSimonian and Laird’s methodology to perform multivariate random effects meta-analyses. Stat Med. 2010;29:1282–97.PubMedCrossRefGoogle Scholar
  53. 53.
    Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21:1539–58.PubMedCrossRefGoogle Scholar
  54. 54.
    Ioannidis JP. Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract. 2008;14:951–7.PubMedCrossRefGoogle Scholar
  55. 55.
    Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629–34.PubMedPubMedCentralCrossRefGoogle Scholar
  56. 56.
    Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med. 2001;20:641–54.PubMedCrossRefGoogle Scholar
  57. 57.
    Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50:1088–101.PubMedCrossRefGoogle Scholar
  58. 58.
    Bürkner PC, Doebler P. Testing for publication bias in diagnostic meta-analysis: a simulation study. Stat Med. 2014;33:3061–77.PubMedCrossRefGoogle Scholar
  59. 59.
    Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32–5.PubMedCrossRefGoogle Scholar
  60. 60.
    Le CT. A solution for the most basic optimization problem associated with an ROC curve. Stat Methods Med Res. 2006;15:571–84.PubMedCrossRefGoogle Scholar
  61. 61.
    Duval S, Tweedie R. A nonparametric ‘Trim and Fill’ method of accounting for publication bias in meta-analysis. JASA. 2000;95:89–98.Google Scholar
  62. 62.
    Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60.PubMedPubMedCentralCrossRefGoogle Scholar
  63. 63.
    Deeks JJ, Higgins JPT, Altman DG, editors. Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Accessed 29 June 2018.
  64. 64.
    Gatsonis C, Paliwal P. Meta-analysis of diagnostic and screening test accuracy evaluations: methodologic primer. AJR Am J Roentgenol. 2006;187:271–81.CrossRefPubMedGoogle Scholar
  65. 65.
    Jackson D, White IR, Riley RD. Quantifying the impact of between-study heterogeneity in multivariate meta-analyses. Stat Med. 2012;31:3805–20.PubMedPubMedCentralCrossRefGoogle Scholar
  66. 66.
    Zhou Y, Dendukuri N. Statistics for quantifying heterogeneity in univariate and bivariate meta-analyses of binary data: The case of meta-analyses of diagnostic accuracy. Stat Med. 2014;33:2701–17.PubMedCrossRefGoogle Scholar
  67. 67.
    Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I2 in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008;8:79.PubMedPubMedCentralCrossRefGoogle Scholar
  68. 68.
    Borenstein M, Higgins JP, Hedges LV, Rothstein HR. Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Res Synth Methods. 2017;8:5–18.PubMedCrossRefGoogle Scholar
  69. 69.
    Böhning D, Malzahn U, Dietz E, Schlattmann P, Viwatwongkasem C, Biggeri A. Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator. Biostatistics. 2002;3:445–57.PubMedCrossRefGoogle Scholar
  70. 70.
    Jackson D, White IR, Riley RD. A matrix-based method of moments for fitting the multivariate random effects model for meta-analysis and meta-regression. Biom J. 2013;55:231–45.PubMedPubMedCentralCrossRefGoogle Scholar
  71. 71.
    Ohle R, McIsaac SM, Woo MY, Perry JJ. Sonography of the optic nerve sheath diameter for detection of raised intracranial pressure compared to computed tomography: a systematic review and meta-analysis. J Ultrasound Med. 2015;34:1285–94.PubMedCrossRefGoogle Scholar
  72. 72.
    Leeflang MM, Moons KG, Reitsma JB, Zwinderman AH. Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clin Chem. 2008;54:729–37.PubMedCrossRefGoogle Scholar
  73. 73.
    Rücker G, Schumacher M. Letter to the editor. Biostatistics. 2009;10:806–7.PubMedCrossRefGoogle Scholar
  74. 74.
    Janda S, Swiston J. Diagnostic accuracy of pleural fluid NT-pro-BNP for pleural effusions of cardiac origin: a systematic review and meta-analysis. BMC Pulm Med. 2010;10:58.PubMedPubMedCentralCrossRefGoogle Scholar
  75. 75.
    Trikalinos TA, Balion CM, Coleman CI, Griffith L, Santaguida PL, Vandermeer B, et al. Chapter 8: meta-analysis of test performance when there is a ‘gold standard’. J Gen Intern Med. 2012;27:S56–66.PubMedCrossRefGoogle Scholar
  76. 76.
    GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ. 2004;328:1490.PubMedCentralCrossRefPubMedGoogle Scholar
  77. 77.
    Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, et al. Rating quality of evidence and strength of recommendations–GRADE: grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008;336:1106–10.PubMedPubMedCentralCrossRefGoogle Scholar
  78. 78.
    Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58:982–90.CrossRefPubMedGoogle Scholar
  79. 79.
    Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Decis Mak. 1993;13:313–21.CrossRefGoogle Scholar
  80. 80.
    Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001;20:2865–84.PubMedPubMedCentralCrossRefGoogle Scholar
  81. 81.
    Lee J, Kim KW, Choi SH, Huh J, Park SH. Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-Part II. Statistical methods of meta-analysis. Korean J Radiol. 2015;16:1188–96.PubMedPubMedCentralCrossRefGoogle Scholar
  82. 82.
    Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007;8:239–51.CrossRefPubMedGoogle Scholar
  83. 83.
    Arends LR, Hamza TH, van Houwelingen JC, Heijenbrok-Kal MH, Hunink MG, Stijnen T. Bivariate random effects meta-analysis of ROC curves. Med Decis Mak. 2008;28:621–38.CrossRefGoogle Scholar
  84. 84.
    Harbord RM, Whiting P, Sterne JA, Egger M, Deeks JJ, Shang A, et al. An empirical comparison of methods for meta-analysis of diagnostic accuracy showed hierarchical models are necessary. J Clin Epidemiol. 2008;61:1095–103.CrossRefPubMedGoogle Scholar
  85. 85.
    Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat Med. 2008;27:625–50.PubMedCrossRefGoogle Scholar
  86. 86.
    Chen Y, Liu Y, Ning J, Nie L, Zhu H, Chu H. A composite likelihood method for bivariate meta-analysis in diagnostic systematic reviews. Stat Methods Med Res. 2014;26:914–30.PubMedPubMedCentralCrossRefGoogle Scholar
  87. 87.
    Chen Y, Hong C, Ning Y, Su X. Meta-analysis of studies with bivariate binary outcomes: a marginal beta-binomial model approach. Stat Med. 2016;35:21–40.PubMedCrossRefGoogle Scholar
  88. 88.
    Guolo A. A double SIMEX approach for bivariate random-effects meta-analysis of diagnostic accuracy studies. BMC Med Res Methodol. 2017;17:6.PubMedPubMedCentralCrossRefGoogle Scholar
  89. 89.
    Rücker G, Schumacher M. Summary ROC curve based on a weighted Youden index for selecting an optimal cutpoint in meta-analysis of diagnostic accuracy. Stat Med. 2010;29:3069–78.PubMedPubMedCentralCrossRefGoogle Scholar
  90. 90.
    Doebler P, Holling H, Böhning D. A mixed model approach to meta-analysis of diagnostic studies with binary test outcome. Psychol Methods. 2012;17:418–36.PubMedPubMedCentralCrossRefGoogle Scholar
  91. 91.
    Holling H, Böhning W, Böhning D. Meta-analysis of diagnostic studies based upon SROC-curves: a mixed model approach using the Lehmann family. Stat Model. 2012;12:347–75.CrossRefGoogle Scholar
  92. 92.
    Charoensawat S, Böhning W, Böhning D, Holling H. Meta-analysis and meta-modelling for diagnostic problems. BMC Med Res Methodol. 2014;14:56.PubMedPubMedCentralCrossRefGoogle Scholar
  93. 93.
    Doebler P, Holling H. Meta-analysis of Diagnostic Accuracy and ROC Curves with Covariate Adjusted Semiparametric mixtures. Psychometrika. 2015;80:1084–104.PubMedPubMedCentralCrossRefGoogle Scholar
  94. 94.
    Schlattmann P, Verba M, Dewey M, Walther M. Mixture models in diagnostic meta-analyses—clustering summary receiver operating characteristic curves accounted for heterogeneity and correlation. J Clin Epidemiol. 2015;68:61–72.PubMedPubMedCentralCrossRefGoogle Scholar
  95. 95.
    Menke J. Bayesian bivariate meta-analysis of sensitivity and specificity: summary of quantitative findings in 50 meta-analyses. J Eval Clin Pract. 2014;20:844–52.PubMedCrossRefGoogle Scholar
  96. 96.
    Dahabreh IJ, Trikalinos TA, Lau J, Schmid C. An empirical assessment of bivariate methods for meta-analysis of test accuracy. Methods Research Reports; Agency for Healthcare Research and Quality (US). 2012;12(13)-EHC136-EF.Google Scholar
  97. 97.
    Novielli N, Cooper NJ, Sutton AJ, Abrams KR. Bayesian model selection for meta-analysis of diagnostic test accuracy data: Application to Ddimer for deep vein thrombosis. Res Synth Methods. 2010;1:226–38.PubMedCrossRefGoogle Scholar
  98. 98.
    Dukic V, Gatsonis C. Meta-analysis of diagnostic test accuracy assessment studies with varying number of thresholds. Biometrics. 2003;59:936–46.CrossRefPubMedGoogle Scholar
  99. 99.
    Putter H, Fiocco M, Stijnen T. Meta-analysis of diagnostic test accuracy studies with multiple thresholds using survival methods. Biom J. 2010;52:95–110.CrossRefPubMedGoogle Scholar
  100. 100.
    Poon WY. A latent normal distribution model for analysing ordinal responses with applications in meta-analysis. Stat Med. 2004;23:2155–72.PubMedCrossRefGoogle Scholar
  101. 101.
    Bipat S, Zwinderman AH, Bossuyt PMM, Stoker J. Multivariate random-effects approach: for meta-analysis of cancer staging studies. Acad Radiol. 2007;14:974–84.PubMedCrossRefGoogle Scholar
  102. 102.
    Hamza TH, Arends LR, van Houwelingen HC, Stijnen T. Multivariate random effects meta-analysis of diagnostic tests with multiple thresholds. BMC Med Res Methodol. 2009;9:73.PubMedPubMedCentralCrossRefGoogle Scholar
  103. 103.
    Riley RD, Takwoingi Y, Trikalinos T, Guha A, Biswas A, Ensor J, et al. Meta-analysis of test accuracy studies with multiple and missing thresholds: a multivariate-normal model. J Biomet Biostat. 2014;5:196.CrossRefGoogle Scholar
  104. 104.
    Steinhauser S, Schumacher M, Rücker G. Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies. BMC Med Res Methodol. 2016;16:97.PubMedPubMedCentralCrossRefGoogle Scholar
  105. 105.
    Martínez-Camblor P. Fully non-parametric receiver operating characteristic curve estimation for random-effects meta-analysis. Stat Methods Med Res. 2017;26:5–20.PubMedPubMedCentralCrossRefGoogle Scholar
  106. 106.
    Hoyer A, Hirt S, Kuss O. Meta-analysis of full ROC curves using bivariate time-to-event models for interval-censored data. Res Syn Meth. 2018;9:62–72.CrossRefGoogle Scholar
  107. 107.
    Chen S, Watson P, Parmigiani G. Accuracy of MSI testing in predicting germline mutations of MSH2 and MLH1: a case study in Bayesian meta-analysis of diagnostic tests without a gold standard. Biostatistics. 2005;6:450–64.PubMedPubMedCentralCrossRefGoogle Scholar
  108. 108.
    Chu H, Nie L, Cole S, Poole C. Meta-analysis of diagnostic accuracy studies accounting for disease prevalence: alternative parameterizations and model selection. Stat Med. 2009;28:2384–99.CrossRefPubMedGoogle Scholar
  109. 109.
    Dendukuri N, Schiller I, Joseph L, Pai M. Bayesian meta-analysis of the accuracy of a test for Tuberculous pleuritis in the absence of a gold standard reference. Biometrics. 2012;68:1285–93.PubMedPubMedCentralCrossRefGoogle Scholar
  110. 110.
    Liu Y, Chen Y, Chu H. A unification of models for meta-analysis of diagnostic accuracy studies without a gold standard. Biometrics. 2015;71:538–47.CrossRefPubMedGoogle Scholar
  111. 111.
    De Groot JA, Dendukuri N, Janssen KJ, Reitsma JB, Brophy J, Joseph L, et al. Adjusting for partial verification or workup bias in meta-analyses of diagnostic accuracy studies. Am J Epidemiol. 2012;175:847–53.PubMedCrossRefGoogle Scholar
  112. 112.
    Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39:207–15.CrossRefPubMedGoogle Scholar
  113. 113.
    Menten J, Boelaert M, Lesaffre E. Bayesian meta-analysis of diagnostic tests allowing for imperfect reference standards. Stat Med. 2013;32:5398–413.PubMedCrossRefGoogle Scholar
  114. 114.
    Ma X, Chen Y, Cole SR, Chu H. A Hybrid Bayesian Hierarchical Model Combining Cohort and Case-control Studies for Meta-analysis of Diagnostic Tests: Accounting for Partial Verification Bias. Stat Methods Med Res. 2016;25:3015–37.CrossRefPubMedGoogle Scholar
  115. 115.
    Li J, Fine JP. Assessing the dependence of sensitivity and specificity on prevalence in meta-analysis. Biostatistics. 2011;12:710–22.PubMedPubMedCentralCrossRefGoogle Scholar
  116. 116.
    Chen Y, Liu Y, Chu H, Ting Lee ML, Schmid CH. A simple and robust method for multivariate meta-analysis of diagnostic test accuracy. Stat Med. 2017;36:105–21.CrossRefPubMedGoogle Scholar
  117. 117.
    Hoyer A, Kuss O. Meta-analysis of diagnostic tests accounting for disease prevalence: a new model using trivariate copulas. Stat Med. 2015;34:1912–24.PubMedPubMedCentralCrossRefGoogle Scholar
  118. 118.
    Takwoingi Y, Guo B, Riley RD, Deeks JJ. Performance of methods for meta-analysis of diagnostic test accuracy with few studies or sparse data. Stat Methods Med Res. 2017;26:1896–911.PubMedPubMedCentralCrossRefGoogle Scholar
  119. 119.
    Riley RD, Dodd SR, Craig JV, Thompson JR, Williamson PR. Meta-analysis of diagnostic test studies using individual patient data and aggregate data. Stat Med. 2008;27:6111–36.CrossRefPubMedGoogle Scholar
  120. 120.
    Ter Riet G, Bachmann LM, Kessels AG, Khan KS. Individual patient data meta-analysis of diagnostic studies: opportunities and challenges. Evid Based Med. 2013;18:165–9.PubMedCrossRefGoogle Scholar
  121. 121.
    Begg CB, Greenes RA, Iglewicz B. The influence of uninterpretability on the assessment of diagnostic tests. J Chronic Dis. 1986;39:575–84.PubMedCrossRefGoogle Scholar
  122. 122.
    Schütz GM, Schlattmann P. Use of 3×2 tables with an intention to diagnose approach to assess clinical performance of diagnostic tests: meta-analytical evaluation of coronary ct angiography studies. BMJ. 2012;345:e6717.CrossRefGoogle Scholar
  123. 123.
    Menke J, Kowalski J. Diagnostic accuracy and utility of coronary CT angiography with consideration of unevaluable results: a systematic review and multivariate Bayesian random-effects meta-analysis with intention to diagnose. Eur Radiol. 2016;26:451–8.PubMedCrossRefGoogle Scholar
  124. 124.
    Ma X, Suri MF, Chu H. A trivariate meta-analysis of diagnostic studies accounting for prevalence and non-evaluable subjects: re-evaluation of the meta-analysis of coronary CT angiography studies. BMC Med Res Methodol. 2014;14:128.PubMedPubMedCentralCrossRefGoogle Scholar
  125. 125.
    McClish DK. Combining and comparing area estimates across studies or strata. Med Decis Mak. 1992;12:274–9.CrossRefGoogle Scholar
  126. 126.
    Kester AD, Buntinx F. Meta-analysis of ROC curves. Med Decis Mak. 2000;20:430–9.CrossRefGoogle Scholar
  127. 127.
    Zwinderman AH, Bossuyt PM. We should not pool diagnostic likelihood ratios in systematic reviews. Stat Med. 2008;27:687–97.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Medical StatisticsUniversity Medical Center GöttingenGöttingenGermany
  2. 2.Department of Medical Biometry and EpidemiologyUniversity Medical Center Hamburg-EppendorfHamburgGermany

Personalised recommendations