Skip to main content
SpringerLink
Log in

The impact of including different study designs in meta-analyses of diagnostic accuracy studies

  • METHODS
  • Published:
European Journal of Epidemiology Aims and scope Submit manuscript

Abstract

Diagnostic accuracy may be overestimated when using certain study designs; thus, the inclusion of studies using different designs in meta-analyses may have important effects on their results, and influence clinical decision making. The main aim of this study was to explore the influence of heterogeneity (based on the inclusion of different study designs) on diagnostic accuracy in a sample of published meta-analyses of diagnostic accuracy studies. We identified 30 systematic reviews which included 95 separate meta-analyses combining the results from a total of 976 individual studies. We classified each individual study according to the study design (case–control studies, clinically relevant patient series or other), and each meta-analysis according to the heterogeneity of the included studies. Furthermore, we registered how the methodological quality of the individual studies was assessed. Finally, for each meta-analysis, the summary measure of diagnostic accuracy was categorised as Good, Fair or Poor. We used logistic regression to assess the relationship between reporting good diagnostic accuracy and heterogeneity. Meta-analyses with heterogeneous populations were over three times more likely to report good diagnostic accuracy compared to meta-analyses that included only clinically relevant patient series (adjusted odds ratio 3.07 95 % CI 1.16–8.11). The combination of studies that use different designs, within the same meta-analysis, may lead to higher estimates of diagnostic accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Glass GV. Primary, secondary and meta-analysis of research. Educ Res. 1976;5:3–8.

    Google Scholar 

  2. Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs. 2003;12:77–84.

    Article  PubMed  Google Scholar 

  3. Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM, Cochrane Diagnostic Test Accuracy Working Group. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149:889–97.

    Article  PubMed  Google Scholar 

  4. Deeks JJ. Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests. Br Med J. 2001;323:157–62.

    Article  CAS  Google Scholar 

  5. Lijmer JG, Bossuyt PM, Heisterkamp SH. Exploring sources of heterogeneity in systematic reviews of diagnostic tests. Stat Med. 2002;21:1525–37.

    Article  PubMed  Google Scholar 

  6. Whiting P, Rutjes AWS, Reitsma JB, Bossuyt PMM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25–37.

    Article  PubMed  Google Scholar 

  7. Lumbreras B, Porta M, Márquez S, Pollán M, Parker LA, Hernandez-Aguado I. QUADOMICS: an adaptation of the Quality Assessment of Diagnostic Accuracy Assessment (QUADAS) for the evaluation of the methodological quality of studies on the diagnostic accuracy of ‘-omics’ based technology. Clin Biochem. 2008;41:1316–25.

    Article  PubMed  Google Scholar 

  8. Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat Med. 2008;27:625–50.

    Article  PubMed  Google Scholar 

  9. Dinnes J, Deeks J, Kirby J, Roderick P. A methodological review of how heterogeneity has been examined in systematic reviews of diagnostic test accuracy. Health Technol Assess. 2005;9:1–113.

    CAS  Google Scholar 

  10. Ioannidis JP, Patsopoulos NA, Rothstein HR. Reasons or excuses for avoiding meta-analysis in forest plots. Br Med J. 2008;336:1413–5.

    Article  Google Scholar 

  11. Whiting P, Harbord R, Kleijnen J. No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Med Res Methodol. 2005;5:19.

    Article  PubMed  Google Scholar 

  12. Berlin JA. Invited commentary: benefits of heterogeneity in meta-analysis of data from epidemiologic studies. Am J Epidemiol. 1995;142:383–7.

    PubMed  CAS  Google Scholar 

  13. Irwig L, Macaskill P, Glasziou P, Fahey M. Meta-analytic methods for diagnostic test accuracy. J Clin Epidemiol. 1995;48:119–30. (discussion 131–132).

    Article  PubMed  CAS  Google Scholar 

  14. Detsky AS, Naylor CD, O’Rourke K, McGeer AJ, L’Abbé KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992;45:255–65.

    Article  PubMed  CAS  Google Scholar 

  15. Tatsioni A, Zarin DA, Aronson N, Samson DJ, Flamm CR, Schmid C, Lau J. Challenges in systematic reviews of diagnostic technologies. Ann Intern Med. 2005;142:1048–55.

    Article  PubMed  Google Scholar 

  16. Hernandez-Aguado I. The winding road towards evidence based diagnoses. J Epidemiol Community Health. 2002;56:323–5.

    Article  PubMed  CAS  Google Scholar 

  17. Irwig L, Bossuyt P, Glasziou P, Gatsonis C, Lijmer J. Designing studies to ensure that estimates of test accuracy are transferable. Br Med J. 2002;324:669–71.

    Article  Google Scholar 

  18. Lijmer JG, Mol BW, Heistekamp S, Bonsel GJ, Prins MH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061–6.

    Article  PubMed  CAS  Google Scholar 

  19. Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;176:469–76.

    Google Scholar 

  20. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189–202.

    Article  PubMed  Google Scholar 

  21. Deville WL, Bezemer PD, Bouter LM. Publications on diagnostic test evaluations in family medicine journals: an optimal search strategy. J Clin Epidemiol. 2000;53:65–9.

    Article  PubMed  CAS  Google Scholar 

  22. Lumbreras B, Parker LA, Porta M, Pollán M, Ioannidis JPA, Hernández-Aguado I. Overinterpretation of clinical applicability in molecular diagnostic research. Clin Chem. 2009;55:786–94.

    Article  PubMed  CAS  Google Scholar 

  23. Lantz CA, Nebenzahl E. Behavior and interpretation of the κ statistic: resolution of the two paradoxes. J Clin Epidemiol. 1996;49:431–4.

    Article  PubMed  CAS  Google Scholar 

  24. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med. 1978;299:926–30.

    Article  PubMed  CAS  Google Scholar 

  25. Whiting P, Rutjes AW, Dinnes J, Reitsma J, Bossuyt PM, Kleijnen J. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess. 2004;8:1–234.

    Google Scholar 

  26. Mallett S, Deeks JJ, Halligan S, Hopewell S, Cornelius V, Altman DG. Systematic reviews of diagnostic tests in cancer: review of methods and reporting. Br Med J. 2006;333:413–20.

    Article  Google Scholar 

  27. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. Br Med J. 2009;339:b2700.

    Article  Google Scholar 

  28. Irwig L, Tosteson AN, Gatsonis C, Lau J, Colditz G, Chalmers TC, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994;120:667–76.

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

This work was supported by the Fundación de Investigación Mutua Madrileña and Centro de Investigació Biomedica en Red en Epidemiología y Salud Pública (CIBERESP).

Conflict of interest

The authors declare that they have no conflict of interest.

Author information

Authors and Affiliations

  1. Public Health Department, Miguel Hernández University, Alicante, Spain

    Lucy A. Parker, Noemí Gómez Saez, Ildefonso Hernández-Aguado & Blanca Lumbreras

  2. CIBER en Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain

    Lucy A. Parker, Miquel Porta, Ildefonso Hernández-Aguado & Blanca Lumbreras

  3. Hospital del Mar Research Institute—IMIM, School of Medicine, Universitat Autònoma de Barcelona, Barcelona, Spain

    Miquel Porta

  4. Departamento de Salud Pública, Historia de La Ciencia y Ginecología, Universidad Miguel Hernandez de Elche, Ctra. Alicante-Valencia, Km 8‘7, Sant Joan d’Alacant, 03550, Alicante, Spain

    Lucy A. Parker

Authors
  1. Lucy A. Parker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Noemí Gómez Saez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miquel Porta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ildefonso Hernández-Aguado
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Blanca Lumbreras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lucy A. Parker.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOC 91 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Parker, L.A., Saez, N.G., Porta, M. et al. The impact of including different study designs in meta-analyses of diagnostic accuracy studies. Eur J Epidemiol 28, 713–720 (2013). https://doi.org/10.1007/s10654-012-9756-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10654-012-9756-9

Keywords

Search

Navigation