Skip to main content

Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative

Abstract

Objectives

To report and analyse the characteristics and performance of the first cohort of Italian radiologists completing the national mammography self-evaluation online test established by the Italian Society of Medical Radiology (SIRM).

Methods

A specifically-built dataset of 132 mammograms (24 with screen-detected cancers and 108 negative cases) was preliminarily tested on 48 radiologists to define pass thresholds (62% sensitivity and 86% specificity) and subsequently made available online to SIRM members during a 13-month timeframe between 2018 and 2019. Associations between participants’ characteristics, pass rates, and diagnostic accuracy were then investigated with descriptive statistics and univariate and multivariable regression analyses.

Results

A total of 342 radiologists completed the test, 151/342 (44.2%) with success. All individual variables, except gender, showed a significant correlation with pass rates and diagnostic sensitivity, confirmed by univariate logistic regression, while only involvement in organised screening programs and number of mammograms read per year showed a positive association with specificity at univariate logistic regression. In the multivariable regression analysis, fewer variables remained significant: > 3000 mammograms read per year for success rate; female gender, public practice setting, and higher experience self-judgement for sensitivity; no variables were significantly associated with specificity.

Conclusions

This national self-evaluation test effectively differentiated multiple aspects of mammographic reading experience, but specific breast imaging experience was shown not to strictly guarantee good diagnostic accuracy. Due to its easy use and the validity of obtained results, this test could be extended to all Italian breast radiologists, regardless of their experience, also as a Breast Unit accreditation criterion.

Key Points

This self-evaluation test was found to be able to differentiate various degrees of mammographic interpretation experience.

Breast cancer screening readers should undergo a self-assessment test, since experience parameters alone do not guarantee diagnostic ability.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

Abbreviations

BI-RADS:

Breast Imaging Reporting and Data System

GISMa:

Gruppo Italiano Screening Mammografico – Italian Group for Mammographic Screening

IQR:

Interquartile range

ONS:

Osservatorio Nazionale Screening

ROI:

Region of interest

SIRM:

Società Italiana di Radiologia Medica e Interventistica – Italian Society of Medical Radiology

References

  1. 1.

    Perry N, Broeders M, de Wolf C, Tornberg S, Holland R, von Karsa L (2007) European guidelines for quality assurance in breast cancer screening and diagnosis. Fourth edition–summary document. Ann Oncol 19:614–622. https://doi.org/10.1093/annonc/mdm481

    Article  PubMed  Google Scholar 

  2. 2.

    Giordano L, Giorgi D, Frigerio A et al (2006) Process indicators and standards for the evaluation of breast cancer screening programmes. Epidemiol Prev 30(5–9):11–47

    Google Scholar 

  3. 3.

    Wilson ARM, Marotti L, Bianchi S et al (2013) The requirements of a specialist Breast Centre. Eur J Cancer 49:3579–3587. https://doi.org/10.1016/j.ejca.2013.07.017

    CAS  Article  PubMed  Google Scholar 

  4. 4.

    Torre LA, Islami F, Siegel RL, Ward EM, Jemal A (2017) Global cancer in women: burden and trends. Cancer Epidemiol Biomarkers Prev 26:444–457. https://doi.org/10.1158/1055-9965.EPI-16-0858

    Article  PubMed  Google Scholar 

  5. 5.

    AIRTUM Working Group (2019) I numeri del cancro in Italia. Associazione Italiana Registri Tumori, Milano. Available via https://www.aiom.it/wp-content/uploads/2019/09/2019_Numeri_Cancro-operatori-web.pdf. Accessed 04 July 2021

  6. 6.

    Lauby-Secretan B, Scoccianti C, Loomis D et al (2015) Breast-Cancer Screening — Viewpoint of the IARC Working Group. N Engl J Med 372:2353–2358. https://doi.org/10.1056/NEJMsr1504363

    CAS  Article  PubMed  Google Scholar 

  7. 7.

    Trimboli RM, Giorgi Rossi P, Battisti NML et al (2020) Do we still need breast cancer screening in the era of targeted therapies and precision medicine? Insights Imaging 11:105. https://doi.org/10.1186/s13244-020-00905-3

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Muratov S, Canelo-Aybar C, Tarride J-E et al (2020) Monitoring and evaluation of breast cancer screening programmes: selecting candidate performance indicators. BMC Cancer 20:795. https://doi.org/10.1186/s12885-020-07289-z

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Giordano L, Castagno R, Giorgi D et al (2015) Breast cancer screening in Italy: evaluating key performance indicators for time trends and activity volumes. Epidemiol Prev 39:30–39

    PubMed  Google Scholar 

  10. 10.

    Giorgi D, Giordano L, Ventura L, Frigerio A, Paci E, Zappa M (2012) Mammography breast cancer screening in Italy: 2010 survey. Epidemiol Prev 36:8–27

    PubMed  Google Scholar 

  11. 11.

    Osservatorio Nazionale Screening (2019) Rapporto 2019. Osservatorio Nazionale Screening, Firenze. Available via https://www.osservatorionazionalescreening.it/content/rapporto. Accessed 04 July 2021

  12. 12.

    Hofvind S, Bennett R, Brisson J et al (2016) Audit feedback on reading performance of screening mammograms: an international comparison. J Med Screen 23:150–159. https://doi.org/10.1177/0969141315610790

    CAS  Article  PubMed  Google Scholar 

  13. 13.

    Rutter CM, Taplin S (2000) Assessing mammographers’ accuracy. J Clin Epidemiol 53:443–450. https://doi.org/10.1016/S0895-4356(99)00218-8

    CAS  Article  PubMed  Google Scholar 

  14. 14.

    Qenam BA, Li T, Tapia K, Brennan PC (2020) The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review. Clin Radiol 75:794.e1-794.e6. https://doi.org/10.1016/j.crad.2020.01.015

    CAS  Article  Google Scholar 

  15. 15.

    Soh BP, Lee W, Kench PL et al (2012) Assessing reader performance in radiology, an imperfect science: lessons from breast screening. Clin Radiol 67:623–628. https://doi.org/10.1016/j.crad.2012.02.007

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Gale AG (2003) Performs: a self-assessment scheme for radiologists in breast screening. Semin Breast Dis 6:148–152. https://doi.org/10.1053/j.sembd.2004.03.006

    Article  Google Scholar 

  17. 17.

    Brennan PC, Trieu PD, Tapia K, Ryan J, Mello-Thoms C, Lee W (2014) BREAST: a novel strategy to improve the detection of breast cancer. In: Fujita H, Hara T, Muramatsu C (eds) Breast imaging. IWDM 2014. Lecture Notes in Computer Science. Springer, Cham.

    Google Scholar 

  18. 18.

    Trieu PDY, Tapia K, Frazer H, Lee W, Brennan P (2019) Improvement of cancer detection on mammograms via BREAST test sets. Acad Radiol 26:e341–e347. https://doi.org/10.1016/j.acra.2018.12.017

    Article  PubMed  Google Scholar 

  19. 19.

    Scott HJ, Gale AG (2006) Breast screening: PERFORMS identifies key mammographic training needs. Br J Radiol 79:S127–S133. https://doi.org/10.1259/bjr/25049149

    Article  PubMed  Google Scholar 

  20. 20.

    Timmers JMH, Verbeek ALM, Pijnappel RM, Broeders MJM, den Heeten GJ (2014) Experiences with a self-test for Dutch breast screening radiologists: lessons learnt. Eur Radiol 24:294–304. https://doi.org/10.1007/s00330-013-3018-4

    CAS  Article  PubMed  Google Scholar 

  21. 21.

    Soh BP, Lee WB, McEntee MF et al (2014) Mammography test sets: reading location and prior images do not affect group performance. Clin Radiol 69:397–402. https://doi.org/10.1016/j.crad.2013.11.010

    CAS  Article  PubMed  Google Scholar 

  22. 22.

    Suleiman WI, Rawashdeh MA, Lewis SJ et al (2016) Impact of breast reader assessment strategy on mammographic radiologists’ test reading performance. J Med Imaging Radiat Oncol 60:352–358. https://doi.org/10.1111/1754-9485.12461

    Article  PubMed  Google Scholar 

  23. 23.

    Scott HJ, Evans A, Gale AG, Murphy A, Reed J (2009) The relationship between real life breast screening and an annual self assessment scheme. In: Sahiner B, Manning DJ (eds) Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment. SPIE, Bellingham, WA, USA

  24. 24.

    Ekpo EU, Alakhras M, Brennan P (2018) Errors in mammography cannot be solved through technology alone. Asian Pacific J Cancer Prev 19:291–301. https://doi.org/10.22034/APJCP.2018.19.2.291

    Article  Google Scholar 

  25. 25.

    Molins E, Macià F, Ferrer F, Maristany M-T, Castells X (2008) Association between radiologists’ experience and accuracy in interpreting screening mammograms. BMC Health Serv Res 8:91. https://doi.org/10.1186/1472-6963-8-91

    Article  PubMed  PubMed Central  Google Scholar 

  26. 26.

    Miglioretti DL, Ichikawa L, Smith RA et al (2017) Correlation between screening mammography interpretive performance on a test set and performance in clinical practice. Acad Radiol 24:1256–1264. https://doi.org/10.1016/j.acra.2017.03.016

    Article  PubMed  PubMed Central  Google Scholar 

  27. 27.

    Soh BP, Lee W, McEntee MF et al (2013) Screening mammography: test set data can reasonably describe actual clinical reporting. Radiology 268:46–53. https://doi.org/10.1148/radiol.13122399

    Article  PubMed  PubMed Central  Google Scholar 

  28. 28.

    Nodine CF, Kundel HL, Mello-Thoms C et al (1999) How experience and training influence mammography expertise. Acad Radiol 6:575–585. https://doi.org/10.1016/S1076-6332(99)80252-9

    CAS  Article  PubMed  Google Scholar 

  29. 29.

    Beam CA, Conant EF, Sickles EA (2003) Association of volume and volume-independent factors with accuracy in screening mammogram interpretation. J Natl Cancer Inst 95:282–290. https://doi.org/10.1093/jnci/95.4.282

    Article  PubMed  Google Scholar 

  30. 30.

    Ciatto S, Ambrogetti D, Morrone D, Del Turco MR (2006) Analysis of the results of a proficiency test in screening mammography at the CSPO of Florence: review of 705 tests. Radiol Med 111:797–803. https://doi.org/10.1007/s11547-006-0073-9

    CAS  Article  PubMed  Google Scholar 

  31. 31.

    Kim SH, Lee EH, Jun JK et al (2019) Interpretive performance and inter-observer agreement on digital mammography test sets. Korean J Radiol 20:218. https://doi.org/10.3348/kjr.2018.0193

    Article  PubMed  Google Scholar 

  32. 32.

    Suleiman WI, Lewis SJ, Georgian-Smith D, Evanoff MG, McEntee MF (2014) Number of mammography cases read per year is a strong predictor of sensitivity. J Med Imaging 1:015503. https://doi.org/10.1117/1.JMI.1.1.015503

    Article  Google Scholar 

  33. 33.

    Elmore JG, Wells CK, Howard DH (1998) Does diagnostic accuracy in mammography depend on radiologists’ experience? J Women’s Heal 7:443–449. https://doi.org/10.1089/jwh.1998.7.443

    CAS  Article  Google Scholar 

  34. 34.

    Esserman L, Cowley H, Eberle C et al (2002) Improving the accuracy of mammography: volume and outcome relationships. J Natl Cancer Inst 94:369–375. https://doi.org/10.1093/jnci/94.5.369

    Article  PubMed  Google Scholar 

  35. 35.

    Rawashdeh MA, Lee WB, Bourne RM et al (2013) Markers of good performance in mammography depend on number of annual readings. Radiology 269:61–67. https://doi.org/10.1148/radiol.13122581

    Article  PubMed  Google Scholar 

  36. 36.

    Reed WM, Lee WB, Cawson JN, Brennan PC (2010) Malignancy detection in digital mammograms. Acad Radiol 17:1409–1413. https://doi.org/10.1016/j.acra.2010.06.016

    Article  PubMed  Google Scholar 

  37. 37.

    Gur D, Bandos AI, Cohen CS et al (2008) The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 249:47–53. https://doi.org/10.1148/radiol.2491072025

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank SIRM for the scientific and logistical support and all SIRM members participating in the survey.

Funding

No funding was received for this study.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Beniamino Brancato.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Dr. Beniamino Brancato.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject of the article.

Statistics and biometry

One of the authors, Dr. Calogero Saieva, has significant statistical expertise.

Informed consent

Written informed consent was obtained from all subjects (radiologists sitting the test) in this study.

Ethical approval

Specific Ethics Committee approval was not required for this study, being it an anonymous survey on fully-deidentified images.

Methodology

• Retrospective

• Diagnostic study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Brancato, B., Peruzzi, F., Saieva, C. et al. Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative. Eur Radiol (2021). https://doi.org/10.1007/s00330-021-08241-w

Download citation

Keywords

  • Breast neoplasms
  • Mammography
  • Mass screening
  • Diagnostic self-evaluation