Skip to main content
SpringerLink
Log in

Citation bias in imaging research: are studies with higher diagnostic accuracy estimates cited more often?

  • Experimental
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To assess the risk of citation bias in imaging diagnostic accuracy research by evaluating whether studies with higher accuracy estimates are cited more frequently than those with lower accuracy estimates.

Methods

We searched Medline for diagnostic accuracy meta-analyses published in imaging journals from January 2005 to April 2016. Primary studies from the meta-analyses were screened; those assessing the diagnostic accuracy of an imaging test and reporting sensitivity and specificity were eligible for inclusion. Studies not indexed in Web of Science, duplicates, and inaccessible articles were excluded. Topic (modality/subspecialty), study design, sample size, journal impact factor, publication date, times cited, sensitivity, and specificity were extracted for each study. Negative binomial regression was performed to evaluate the association of citation rate (times cited per month since publication) with Youden’s index (sensitivity + specificity -1), highest sensitivity, and highest specificity, controlling for the potential confounding effects of modality, subspecialty, impact factor, study design, sample size, and source meta-analysis.

Results

There were 1016 primary studies included. A positive association between Youden’s index and citation rate was present, with a regression coefficient of 0.33 (p = 0.016). The regression coefficient for sensitivity was 0.41 (p = 0.034), and for specificity, 0.32 (p = 0.15).

Conclusion

A positive association exists between diagnostic accuracy estimates and citation rates, indicating that there is evidence of citation bias in imaging diagnostic accuracy literature. Overestimation of imaging test accuracy may contribute to patient harm from incorrect interpretation of test results.

Key Points

• Studies with higher accuracy estimates may be cited more frequently than those with lower accuracy estimates.

• This citation bias could lead clinicians, reviews, and clinical practice guidelines to overestimate the accuracy of imaging tests, contributing to patient harm from incorrect interpretation of test results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Abbreviations

CR:

Conventional radiography

CT:

Computed tomography

DTA:

Diagnostic test accuracy

ID:

Identification

MM:

Mammography

MR:

Magnetic resonance imaging

NM/O:

Nuclear medicine/other

PET:

Positron emission tomography

Sens:

Sensitivity

Spec:

Specificity

US:

Ultrasound

References

  1. Duyx B, Urlings MJE, Swaen GMH, Bouter LM, Zeegers MP (2017) Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol 88:92–101

    Article  PubMed  Google Scholar 

  2. Jannot AS, Agoritsas T, Gayet-Ageron A, Perneger TV (2013) Citation bias favoring statistically significant studies was present in medical research. J Clin Epidemiol 66:296–301

    Article  PubMed  Google Scholar 

  3. Greenberg SA (2009) How citation distortions create unfounded authority: analysis of a citation network. BMJ 339:b2680

    Article  PubMed  PubMed Central  Google Scholar 

  4. Seglen PO (1997) Citations and journal impact factors: questionable indicators of research quality. Allergy 52:1050–1056

    Article  CAS  PubMed  Google Scholar 

  5. Steneck NH (2006) Fostering integrity in research: definitions, current knowledge, and future directions. Sci Eng Ethics 12:53–74

    Article  PubMed  Google Scholar 

  6. Hutchison BG, Oxman AD, Lloyd S (1995) Comprehensiveness and bias in reporting clinical trials. Study of reviews of pneumococcal vaccine effectiveness. Can Fam Physician 41:1356–1360

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Ravnskov U (1992) Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 305:15–19

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Misemer BS, Platts-Mills TF, Jones CW (2016) Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: a cross-sectional analysis. Trials 17:473

    Article  PubMed  PubMed Central  Google Scholar 

  9. Kjaergard LL, Gluud C (2002) Citation bias of hepato-biliary randomized clinical trials. J Clin Epidemiol 55:407–410

    Article  PubMed  Google Scholar 

  10. Korevaar DA, Cohen JF, Spijker R et al (2016) Reported estimates of diagnostic accuracy in ophthalmology conference abstracts were not associated with full-text publication. J Clin Epidemiol 79:96–103

    Article  PubMed  PubMed Central  Google Scholar 

  11. Korevaar DA, van Es N, Zwinderman AH, Cohen JF, Bossuyt PM (2016) Time to publication among completed diagnostic accuracy studies: associated with reported accuracy estimates. BMC Med Res Methodol 16:68

    Article  PubMed  PubMed Central  Google Scholar 

  12. Sharifabadi AD, Korevaar DA, McGrath TM, et al (2018) Reporting bias in imaging: higher accuracy is linked to faster publication. Eur Radiol 28:3632−3639

  13. Frank RA, McInnes MDF (2018) Citation bias in imaging research: are studies with higher diagnostic accuracy estimates cited more often? Study protocol. Retrieved from osf.io/j964x

  14. Frank RA, McInnes MDF, Levine D et al (2017) Are study and journal characteristics reliable indicators of ‘truth’ in imaging research? Radiology 287:215–223

    Article  PubMed  Google Scholar 

  15. McGrath TA, McInnes MD, Korevaar DA, Bossuyt PM (2016) Meta-analyses of diagnostic accuracy in imaging journals: analysis of pooling techniques and their effect on summary estimates of diagnostic accuracy. Radiology 281:78–85

    Article  PubMed  Google Scholar 

  16. van Enst WA, Ochodo E, Scholten RJ, Hooft L, Leeflang MM (2014) Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study. BMC Med Res Methodol 14:70

    Article  PubMed  PubMed Central  Google Scholar 

  17. Devillé WL, Bezemer PD, Bouter LM (2000) Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol 53:65–69

    Article  PubMed  Google Scholar 

  18. Reuters T Science citation index expanded-radiology, nuclear medicine & medical imaging-journal list. Available via: http://science.thomsonreuters.com.proxy.bib.uottawa.ca/cgi-bin/jrnlst/jlresults.cgi?PC=D&SC=VY. Accessed 15 Apr 2015

  19. R Core Team (2017) R: A language and environment forstatistical computing. R Foundation for Statistical Computing, V., Austria. Available via https://www.R-project.org/. Accessed 1 Aug 2018

  20. Nieminen P, Carpenter J, Rucker G, Schumacher M (2006) The relationship between quality of research and citation frequency. BMC Med Res Methodol 6:42

    Article  PubMed  PubMed Central  Google Scholar 

  21. Kulkarni AV, Busse JW, Shams I (2007) Characteristics associated with citation rate of the medical literature. PLoS One 2:403

    Article  Google Scholar 

  22. Sawin VI, Robinson KA (2016) Biased and inadequate citation of prior research in reports of cardiovascular trials is a continuing source of waste in research. J Clin Epidemiol 69:174–178

    Article  PubMed  Google Scholar 

  23. Sheth U, Simunovic N, Tornetta P 3rd, Einhorn TA, Bhandari M (2011) Poor citation of prior evidence in hip fracture trials. J Bone Joint Surg Am 93:2079–2086

  24. Etter JF, Stapleton J (2009) Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals. J Clin Epidemiol 62:831–837

    Article  PubMed  Google Scholar 

  25. Nieminen P, Rucker G, Miettunen J, Carpenter J, Schumacher M (2007) Statistically significant papers in psychiatry were cited more often than others. J Clin Epidemiol 60:939–946

    Article  PubMed  Google Scholar 

  26. Saha S, Saint A, Christakis DA (2003) Impact factor: a valid measure of journal quality? J Med Libr Assoc 9:42–46

    Google Scholar 

  27. Dilauro M, McInnes MD, Korevaar DA et al (2016) Is there an association between STARD statement adherence and citation rate? Radiology 280:62–67

  28. Choi YJ, Chung MS, Koo HJ, Park JE, Yoon HM, Park SH (2016) Does the reporting quality of diagnostic test accuracy studies, as defined by STARD 2015, affect citation? Korean J Radiol 17:706–714

    Article  PubMed  PubMed Central  Google Scholar 

  29. McInnes MDF, Moher D, Thombs BD et al (2018) Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement. JAMA 319:388–396

    Article  Google Scholar 

  30. McGrath TA, McInnes MDF, van Es N, Leeflang MMG, Korevaar DA, Bossuyt PMM (2017) Overinterpretation of research findings: evidence of “spin” in systematic reviews of diagnostic accuracy studies. Clin Chem 63:1353–1362

    Article  PubMed  Google Scholar 

  31. Ochodo EA, de Haan MC, Reitsma JB, Hooft L, Bossuyt PM, Leeflang MM (2013) Overinterpretation and misreporting of diagnostic accuracy studies: evidence of “spin”. Radiology 267:581–588

    Article  PubMed  Google Scholar 

Download references

Funding

This study has received funding by the University of Ottawa, Department of Radiology Research Stipend Program.

Author information

Authors and Affiliations

  1. Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada

    Robert A. Frank, Anahita Dehmoobad Sharifabadi, Trevor A. McGrath, Wilfred Dang, Nicole Li, Isabelle D. Gauthier & Mark Z. Wu

  2. Department of Radiology, University of Ottawa, Ottawa, ON, Canada

    Robert A. Frank, Anahita Dehmoobad Sharifabadi, Jean-Paul Salameh, Trevor A. McGrath, Wilfred Dang, Nicole Li, Isabelle D. Gauthier, Mark Z. Wu & Matthew D. F. McInnes

  3. Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada

    Jean-Paul Salameh & Matthew D. F. McInnes

  4. Clinical Epidemiology and Biostatistics and Bioinformatics, Amsterdam Public Health Research Institute, Academic Medical Center, University of Amsterdam, Amsterdam, Netherlands

    Noémie Kraaijpoel & Patrick M. Bossuyt

  5. Department of Vascular Medicine, Academic Medical Center, Amsterdam, the Netherlands

    Noémie Kraaijpoel

  6. Department of Radiology, Beth Israel Deaconess Medical Center, Boston, MA, USA

    Deborah Levine

Authors
  1. Robert A. Frank
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anahita Dehmoobad Sharifabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean-Paul Salameh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Trevor A. McGrath
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Noémie Kraaijpoel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wilfred Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nicole Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Isabelle D. Gauthier
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mark Z. Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Patrick M. Bossuyt
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Deborah Levine
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Matthew D. F. McInnes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthew D. F. McInnes.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Matthew McInnes.

Conflict of interest

The authors declare that they have no conflict of interest.

Statistics and biometry

One of the authors has significant statistical expertise (Drs McInnes and Bossuyt).

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was not required because this study is an evaluation of published literature.

Methodology

• Retrospective

• Cross sectional study

• Multicenter study

Electronic supplementary material

ESM 1

(PDF 148 kb)

ESM 2

(DOCX 5683 kb)

ESM 3

(XLSX 33 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Frank, R.A., Sharifabadi, A.D., Salameh, JP. et al. Citation bias in imaging research: are studies with higher diagnostic accuracy estimates cited more often?. Eur Radiol 29, 1657–1664 (2019). https://doi.org/10.1007/s00330-018-5801-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-018-5801-8

Keywords

Search

Navigation