Abstract
This work highlights concepts on the potential for design-related factors to bias estimates of test accuracy in comparative imaging research. We chose two design factors, selection of eligible subjects and the reference standard, to examine the effect of design limitations on estimates of accuracy. Estimates of sensitivity in a study of the comparative accuracy of mammography and ultrasound differed according to how subjects were selected. Comparison of a new imaging test with an existing test should distinguish whether the new test is to be used as a replacement for, or as an adjunct to, the conventional test, to guide the method for subject selection. Quality of the reference standard, examined in a meta-analysis of preoperative breast MRI, varied across studies and was associated with estimates of incremental accuracy. Potential solutions to deal with the reference standard are outlined where an ideal reference standard may not be available in all subjects. These examples of breast imaging research demonstrate that design-related bias, when comparing a new imaging test with a conventional imaging test, may bias accuracy in a direction that favours the new test by overestimating the accuracy of the new test or by underestimating that of the conventional test.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bossuyt PM, Reitsma JB, Bruns DE et al (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative. Radiology 226:24–28
Sardanelli F (2010) Evidence-based radiology: why and how? Eur Radiol 20:1–15
Lord SJ, Irwig L, Simes RJ (2006) When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Ann Intern Med 144:850–855
Lijmer JG, Bossuyt PM (2009) Various randomized designs can be used to evaluate medical tests. J Clin Epidemiol 62:364–373
Schünemann HJ, Oxman AD, Brozek J et al (2008) GRADE: grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ 336:1106–1110
Bossuyt PM, Irwig L, Craig J, Glasziou P (2006) Comparative accuracy: assessing new tests against existing diagnostic pathways. BMJ 332:1089–1092
Lord SJ, Irwig L, Bossuyt PM (2009) Using the principles of randomized controlled trial design to guide test evaluation. Med Decis Making 29:E1–E12
Reitsma JB, Rutjes AWS, Whiting P et al (2009) Assessing methodological quality. In: Deeks JJ, Bossuyt PM, Gatsonis C (Eds.), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0.0. The Cochrane Collaboration 2009. Available via The Cochrane Collaboration. http://srdta.cochrane.org. Accessed 6 February 2010
Irwig L, Bossuyt PM, Glasziou P et al (2002) Designing studies to ensure that estimates of test accuracy are transferable. BMJ 324:669–671
Houssami N, Ciatto S, Irwig L et al (2002) The comparative sensitivity of mammography and ultrasound in women with breast symptoms: an age-specific analysis. Breast 11:125–130
Reitsma JB, Rutjes AW, Khan KS et al (2009) A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol 62:797–806
Houssami N, Ciatto S, Macaskill P et al (2008) Accuracy and surgical impact of magnetic resonance imaging in breast cancer staging: systematic review and meta-analysis in detection of multifocal and multicentric cancer. J Clin Oncol 26:3248–3258
Leeflang M, Reitsma J, Scholten R et al (2007) Impact of adjustment for quality on results of metaanalyses of diagnostic accuracy. Clin Chem 53:164–172
Sardanelli F, Giuseppetti GM, Panizza P et al (2004) Italian Trial for Breast MR in Multifocal/Multicentric Cancer. Sensitivity of MRI versus mammography for detecting foci of multifocal, multicentric breast cancer in fatty and dense breasts using the wholebreast pathologic examination as a gold standard. AJR Am J Roentgenol 183:1149–1157
Lehman CD, Gatsonis C, Kuhl CK et al (2007) MRI evaluation of the contralateral breast in women with recently diagnosed breast cancer. N Engl J Med 356:1295–1303
Houssami N, Irwig L, Simpson JM et al (2003) Sydney Breast Imaging Accuracy Study: comparative sensitivity and specificity of mammography and sonography in young women with symptoms. AJR Am J Roentgenol 180:935–940
Berg WA, Gutierrez L, NessAiver MS et al (2004) Diagnostic accuracy of mammography, clinical examination, US, and MR imaging in preoperative assessment of breast cancer. Radiology 233:830–849
Turnbull LW, Brown SR, Olivier C et al (2010) Multicentre randomised controlled trial examining the cost-effectiveness of contrast-enhanced high field magnetic resonance imaging in women with primary breast cancer scheduled for wide local excision (COMICE) Health Technol Assess 14. Available via http://www.hta.ac.uk/1216. Accessed 10 February 2010
Whiting P, Rutjes AWS, Reitsma JB et al (2004) Sources of variation and bias in studies of diagnostic accuracy. Ann Intern Med 140:189–202
Houssami N, Irwig L, Simpson JM et al (2004) The influence of clinical information on the accuracy of diagnostic mammography. Breast Cancer Res Treat 121:1–6
Houssami N, Irwig L, Simpson S et al (2005) The influence of knowledge of mammography findings on the accuracy of breast ultrasound in symptomatic women. Breast J 11:167–172
Lijmer JG, Mol BW, Heisterkamp S et al (1999) Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 282:1061–1066
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Houssami, N., Ciatto, S. Design-related bias in estimates of accuracy when comparing imaging tests: examples from breast imaging research. Eur Radiol 20, 2061–2066 (2010). https://doi.org/10.1007/s00330-010-1779-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00330-010-1779-6