To report and analyse the characteristics and performance of the first cohort of Italian radiologists completing the national mammography self-evaluation online test established by the Italian Society of Medical Radiology (SIRM).
A specifically-built dataset of 132 mammograms (24 with screen-detected cancers and 108 negative cases) was preliminarily tested on 48 radiologists to define pass thresholds (62% sensitivity and 86% specificity) and subsequently made available online to SIRM members during a 13-month timeframe between 2018 and 2019. Associations between participants’ characteristics, pass rates, and diagnostic accuracy were then investigated with descriptive statistics and univariate and multivariable regression analyses.
A total of 342 radiologists completed the test, 151/342 (44.2%) with success. All individual variables, except gender, showed a significant correlation with pass rates and diagnostic sensitivity, confirmed by univariate logistic regression, while only involvement in organised screening programs and number of mammograms read per year showed a positive association with specificity at univariate logistic regression. In the multivariable regression analysis, fewer variables remained significant: > 3000 mammograms read per year for success rate; female gender, public practice setting, and higher experience self-judgement for sensitivity; no variables were significantly associated with specificity.
This national self-evaluation test effectively differentiated multiple aspects of mammographic reading experience, but specific breast imaging experience was shown not to strictly guarantee good diagnostic accuracy. Due to its easy use and the validity of obtained results, this test could be extended to all Italian breast radiologists, regardless of their experience, also as a Breast Unit accreditation criterion.
• This self-evaluation test was found to be able to differentiate various degrees of mammographic interpretation experience.
• Breast cancer screening readers should undergo a self-assessment test, since experience parameters alone do not guarantee diagnostic ability.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Breast Imaging Reporting and Data System
Gruppo Italiano Screening Mammografico – Italian Group for Mammographic Screening
Osservatorio Nazionale Screening
Region of interest
Società Italiana di Radiologia Medica e Interventistica – Italian Society of Medical Radiology
Perry N, Broeders M, de Wolf C, Tornberg S, Holland R, von Karsa L (2007) European guidelines for quality assurance in breast cancer screening and diagnosis. Fourth edition–summary document. Ann Oncol 19:614–622. https://doi.org/10.1093/annonc/mdm481
Giordano L, Giorgi D, Frigerio A et al (2006) Process indicators and standards for the evaluation of breast cancer screening programmes. Epidemiol Prev 30(5–9):11–47
Wilson ARM, Marotti L, Bianchi S et al (2013) The requirements of a specialist Breast Centre. Eur J Cancer 49:3579–3587. https://doi.org/10.1016/j.ejca.2013.07.017
Torre LA, Islami F, Siegel RL, Ward EM, Jemal A (2017) Global cancer in women: burden and trends. Cancer Epidemiol Biomarkers Prev 26:444–457. https://doi.org/10.1158/1055-9965.EPI-16-0858
AIRTUM Working Group (2019) I numeri del cancro in Italia. Associazione Italiana Registri Tumori, Milano. Available via https://www.aiom.it/wp-content/uploads/2019/09/2019_Numeri_Cancro-operatori-web.pdf. Accessed 04 July 2021
Lauby-Secretan B, Scoccianti C, Loomis D et al (2015) Breast-Cancer Screening — Viewpoint of the IARC Working Group. N Engl J Med 372:2353–2358. https://doi.org/10.1056/NEJMsr1504363
Trimboli RM, Giorgi Rossi P, Battisti NML et al (2020) Do we still need breast cancer screening in the era of targeted therapies and precision medicine? Insights Imaging 11:105. https://doi.org/10.1186/s13244-020-00905-3
Muratov S, Canelo-Aybar C, Tarride J-E et al (2020) Monitoring and evaluation of breast cancer screening programmes: selecting candidate performance indicators. BMC Cancer 20:795. https://doi.org/10.1186/s12885-020-07289-z
Giordano L, Castagno R, Giorgi D et al (2015) Breast cancer screening in Italy: evaluating key performance indicators for time trends and activity volumes. Epidemiol Prev 39:30–39
Giorgi D, Giordano L, Ventura L, Frigerio A, Paci E, Zappa M (2012) Mammography breast cancer screening in Italy: 2010 survey. Epidemiol Prev 36:8–27
Osservatorio Nazionale Screening (2019) Rapporto 2019. Osservatorio Nazionale Screening, Firenze. Available via https://www.osservatorionazionalescreening.it/content/rapporto. Accessed 04 July 2021
Hofvind S, Bennett R, Brisson J et al (2016) Audit feedback on reading performance of screening mammograms: an international comparison. J Med Screen 23:150–159. https://doi.org/10.1177/0969141315610790
Rutter CM, Taplin S (2000) Assessing mammographers’ accuracy. J Clin Epidemiol 53:443–450. https://doi.org/10.1016/S0895-4356(99)00218-8
Qenam BA, Li T, Tapia K, Brennan PC (2020) The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review. Clin Radiol 75:794.e1-794.e6. https://doi.org/10.1016/j.crad.2020.01.015
Soh BP, Lee W, Kench PL et al (2012) Assessing reader performance in radiology, an imperfect science: lessons from breast screening. Clin Radiol 67:623–628. https://doi.org/10.1016/j.crad.2012.02.007
Gale AG (2003) Performs: a self-assessment scheme for radiologists in breast screening. Semin Breast Dis 6:148–152. https://doi.org/10.1053/j.sembd.2004.03.006
Brennan PC, Trieu PD, Tapia K, Ryan J, Mello-Thoms C, Lee W (2014) BREAST: a novel strategy to improve the detection of breast cancer. In: Fujita H, Hara T, Muramatsu C (eds) Breast imaging. IWDM 2014. Lecture Notes in Computer Science. Springer, Cham.
Trieu PDY, Tapia K, Frazer H, Lee W, Brennan P (2019) Improvement of cancer detection on mammograms via BREAST test sets. Acad Radiol 26:e341–e347. https://doi.org/10.1016/j.acra.2018.12.017
Scott HJ, Gale AG (2006) Breast screening: PERFORMS identifies key mammographic training needs. Br J Radiol 79:S127–S133. https://doi.org/10.1259/bjr/25049149
Timmers JMH, Verbeek ALM, Pijnappel RM, Broeders MJM, den Heeten GJ (2014) Experiences with a self-test for Dutch breast screening radiologists: lessons learnt. Eur Radiol 24:294–304. https://doi.org/10.1007/s00330-013-3018-4
Soh BP, Lee WB, McEntee MF et al (2014) Mammography test sets: reading location and prior images do not affect group performance. Clin Radiol 69:397–402. https://doi.org/10.1016/j.crad.2013.11.010
Suleiman WI, Rawashdeh MA, Lewis SJ et al (2016) Impact of breast reader assessment strategy on mammographic radiologists’ test reading performance. J Med Imaging Radiat Oncol 60:352–358. https://doi.org/10.1111/1754-9485.12461
Scott HJ, Evans A, Gale AG, Murphy A, Reed J (2009) The relationship between real life breast screening and an annual self assessment scheme. In: Sahiner B, Manning DJ (eds) Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment. SPIE, Bellingham, WA, USA
Ekpo EU, Alakhras M, Brennan P (2018) Errors in mammography cannot be solved through technology alone. Asian Pacific J Cancer Prev 19:291–301. https://doi.org/10.22034/APJCP.2018.19.2.291
Molins E, Macià F, Ferrer F, Maristany M-T, Castells X (2008) Association between radiologists’ experience and accuracy in interpreting screening mammograms. BMC Health Serv Res 8:91. https://doi.org/10.1186/1472-6963-8-91
Miglioretti DL, Ichikawa L, Smith RA et al (2017) Correlation between screening mammography interpretive performance on a test set and performance in clinical practice. Acad Radiol 24:1256–1264. https://doi.org/10.1016/j.acra.2017.03.016
Soh BP, Lee W, McEntee MF et al (2013) Screening mammography: test set data can reasonably describe actual clinical reporting. Radiology 268:46–53. https://doi.org/10.1148/radiol.13122399
Nodine CF, Kundel HL, Mello-Thoms C et al (1999) How experience and training influence mammography expertise. Acad Radiol 6:575–585. https://doi.org/10.1016/S1076-6332(99)80252-9
Beam CA, Conant EF, Sickles EA (2003) Association of volume and volume-independent factors with accuracy in screening mammogram interpretation. J Natl Cancer Inst 95:282–290. https://doi.org/10.1093/jnci/95.4.282
Ciatto S, Ambrogetti D, Morrone D, Del Turco MR (2006) Analysis of the results of a proficiency test in screening mammography at the CSPO of Florence: review of 705 tests. Radiol Med 111:797–803. https://doi.org/10.1007/s11547-006-0073-9
Kim SH, Lee EH, Jun JK et al (2019) Interpretive performance and inter-observer agreement on digital mammography test sets. Korean J Radiol 20:218. https://doi.org/10.3348/kjr.2018.0193
Suleiman WI, Lewis SJ, Georgian-Smith D, Evanoff MG, McEntee MF (2014) Number of mammography cases read per year is a strong predictor of sensitivity. J Med Imaging 1:015503. https://doi.org/10.1117/1.JMI.1.1.015503
Elmore JG, Wells CK, Howard DH (1998) Does diagnostic accuracy in mammography depend on radiologists’ experience? J Women’s Heal 7:443–449. https://doi.org/10.1089/jwh.1998.7.443
Esserman L, Cowley H, Eberle C et al (2002) Improving the accuracy of mammography: volume and outcome relationships. J Natl Cancer Inst 94:369–375. https://doi.org/10.1093/jnci/94.5.369
Rawashdeh MA, Lee WB, Bourne RM et al (2013) Markers of good performance in mammography depend on number of annual readings. Radiology 269:61–67. https://doi.org/10.1148/radiol.13122581
Reed WM, Lee WB, Cawson JN, Brennan PC (2010) Malignancy detection in digital mammograms. Acad Radiol 17:1409–1413. https://doi.org/10.1016/j.acra.2010.06.016
Gur D, Bandos AI, Cohen CS et al (2008) The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 249:47–53. https://doi.org/10.1148/radiol.2491072025
We thank SIRM for the scientific and logistical support and all SIRM members participating in the survey.
No funding was received for this study.
The scientific guarantor of this publication is Dr. Beniamino Brancato.
Conflict of interest
The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject of the article.
Statistics and biometry
One of the authors, Dr. Calogero Saieva, has significant statistical expertise.
Written informed consent was obtained from all subjects (radiologists sitting the test) in this study.
Specific Ethics Committee approval was not required for this study, being it an anonymous survey on fully-deidentified images.
• Diagnostic study
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Brancato, B., Peruzzi, F., Saieva, C. et al. Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative. Eur Radiol (2021). https://doi.org/10.1007/s00330-021-08241-w
- Breast neoplasms
- Mass screening
- Diagnostic self-evaluation