European Radiology

, Volume 20, Issue 9, pp 2061–2066 | Cite as

Design-related bias in estimates of accuracy when comparing imaging tests: examples from breast imaging research

  • Nehmat HoussamiEmail author
  • Stefano Ciatto


This work highlights concepts on the potential for design-related factors to bias estimates of test accuracy in comparative imaging research. We chose two design factors, selection of eligible subjects and the reference standard, to examine the effect of design limitations on estimates of accuracy. Estimates of sensitivity in a study of the comparative accuracy of mammography and ultrasound differed according to how subjects were selected. Comparison of a new imaging test with an existing test should distinguish whether the new test is to be used as a replacement for, or as an adjunct to, the conventional test, to guide the method for subject selection. Quality of the reference standard, examined in a meta-analysis of preoperative breast MRI, varied across studies and was associated with estimates of incremental accuracy. Potential solutions to deal with the reference standard are outlined where an ideal reference standard may not be available in all subjects. These examples of breast imaging research demonstrate that design-related bias, when comparing a new imaging test with a conventional imaging test, may bias accuracy in a direction that favours the new test by overestimating the accuracy of the new test or by underestimating that of the conventional test.


Accuracy Breast imaging Comparative design Selection bias Reference standard 


  1. 1.
    Bossuyt PM, Reitsma JB, Bruns DE et al (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative. Radiology 226:24–28CrossRefPubMedGoogle Scholar
  2. 2.
    Sardanelli F (2010) Evidence-based radiology: why and how? Eur Radiol 20:1–15CrossRefPubMedGoogle Scholar
  3. 3.
    Lord SJ, Irwig L, Simes RJ (2006) When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Ann Intern Med 144:850–855PubMedGoogle Scholar
  4. 4.
    Lijmer JG, Bossuyt PM (2009) Various randomized designs can be used to evaluate medical tests. J Clin Epidemiol 62:364–373CrossRefPubMedGoogle Scholar
  5. 5.
    Schünemann HJ, Oxman AD, Brozek J et al (2008) GRADE: grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ 336:1106–1110CrossRefPubMedGoogle Scholar
  6. 6.
    Bossuyt PM, Irwig L, Craig J, Glasziou P (2006) Comparative accuracy: assessing new tests against existing diagnostic pathways. BMJ 332:1089–1092CrossRefPubMedGoogle Scholar
  7. 7.
    Lord SJ, Irwig L, Bossuyt PM (2009) Using the principles of randomized controlled trial design to guide test evaluation. Med Decis Making 29:E1–E12CrossRefPubMedGoogle Scholar
  8. 8.
    Reitsma JB, Rutjes AWS, Whiting P et al (2009) Assessing methodological quality. In: Deeks JJ, Bossuyt PM, Gatsonis C (Eds.), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0.0. The Cochrane Collaboration 2009. Available via The Cochrane Collaboration. Accessed 6 February 2010
  9. 9.
    Irwig L, Bossuyt PM, Glasziou P et al (2002) Designing studies to ensure that estimates of test accuracy are transferable. BMJ 324:669–671CrossRefPubMedGoogle Scholar
  10. 10.
    Houssami N, Ciatto S, Irwig L et al (2002) The comparative sensitivity of mammography and ultrasound in women with breast symptoms: an age-specific analysis. Breast 11:125–130CrossRefPubMedGoogle Scholar
  11. 11.
    Reitsma JB, Rutjes AW, Khan KS et al (2009) A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol 62:797–806CrossRefPubMedGoogle Scholar
  12. 12.
    Houssami N, Ciatto S, Macaskill P et al (2008) Accuracy and surgical impact of magnetic resonance imaging in breast cancer staging: systematic review and meta-analysis in detection of multifocal and multicentric cancer. J Clin Oncol 26:3248–3258CrossRefPubMedGoogle Scholar
  13. 13.
    Leeflang M, Reitsma J, Scholten R et al (2007) Impact of adjustment for quality on results of metaanalyses of diagnostic accuracy. Clin Chem 53:164–172CrossRefPubMedGoogle Scholar
  14. 14.
    Sardanelli F, Giuseppetti GM, Panizza P et al (2004) Italian Trial for Breast MR in Multifocal/Multicentric Cancer. Sensitivity of MRI versus mammography for detecting foci of multifocal, multicentric breast cancer in fatty and dense breasts using the wholebreast pathologic examination as a gold standard. AJR Am J Roentgenol 183:1149–1157PubMedGoogle Scholar
  15. 15.
    Lehman CD, Gatsonis C, Kuhl CK et al (2007) MRI evaluation of the contralateral breast in women with recently diagnosed breast cancer. N Engl J Med 356:1295–1303CrossRefPubMedGoogle Scholar
  16. 16.
    Houssami N, Irwig L, Simpson JM et al (2003) Sydney Breast Imaging Accuracy Study: comparative sensitivity and specificity of mammography and sonography in young women with symptoms. AJR Am J Roentgenol 180:935–940PubMedGoogle Scholar
  17. 17.
    Berg WA, Gutierrez L, NessAiver MS et al (2004) Diagnostic accuracy of mammography, clinical examination, US, and MR imaging in preoperative assessment of breast cancer. Radiology 233:830–849CrossRefPubMedGoogle Scholar
  18. 18.
    Turnbull LW, Brown SR, Olivier C et al (2010) Multicentre randomised controlled trial examining the cost-effectiveness of contrast-enhanced high field magnetic resonance imaging in women with primary breast cancer scheduled for wide local excision (COMICE) Health Technol Assess 14. Available via Accessed 10 February 2010
  19. 19.
    Whiting P, Rutjes AWS, Reitsma JB et al (2004) Sources of variation and bias in studies of diagnostic accuracy. Ann Intern Med 140:189–202PubMedGoogle Scholar
  20. 20.
    Houssami N, Irwig L, Simpson JM et al (2004) The influence of clinical information on the accuracy of diagnostic mammography. Breast Cancer Res Treat 121:1–6CrossRefGoogle Scholar
  21. 21.
    Houssami N, Irwig L, Simpson S et al (2005) The influence of knowledge of mammography findings on the accuracy of breast ultrasound in symptomatic women. Breast J 11:167–172CrossRefPubMedGoogle Scholar
  22. 22.
    Lijmer JG, Mol BW, Heisterkamp S et al (1999) Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 282:1061–1066CrossRefPubMedGoogle Scholar

Copyright information

© European Society of Radiology 2010

Authors and Affiliations

  1. 1.School of Public HealthUniversity of SydneySydneyAustralia
  2. 2.Breast Screening ProgramVeronaItaly

Personalised recommendations