Skip to main content
SpringerLink
Log in

Experiences with a self-test for Dutch breast screening radiologists: lessons learnt

  • Breast
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Purpose

To evaluate a self-test for Dutch breast screening radiologists introduced as part of the national quality assurance programme.

Methods and materials

A total of 144 radiologists were invited to complete a test-set of 60 screening mammograms (20 malignancies). Participants assigned findings such as location, lesion type and BI-RADS. We determined areas under the receiver operating characteristics (ROC) curves (AUC), case and lesion sensitivity and specificity, agreement (kappa) and correlation between reader characteristics and case sensitivity (Spearman correlation coefficients).

Results

A total of 110 radiologists completed the test (76 %). Participants read a median number of 10,000 screening mammograms/year. Median AUC value was 0.93, case and lesion sensitivity was 91 % and case specificity 94 %. We found substantial agreement for recall (κ = 0.77) and laterality (κ = 0.80), moderate agreement for lesion type (κ = 0.57) and BI-RADS (κ = 0.45) and no correlation between case sensitivity and reader characteristics.

Conclusion

Areas under the ROC curve, case sensitivity and lesion sensitivity were satisfactory and recall agreement was substantial. However, agreement in lesion type and BI-RADS could be improved; further education might be aimed at reducing interobserver variation in interpretation and description of abnormalities. We offered individual feedback on interpretive performance and overall feedback at group level. Future research will determine whether performance has improved.

Key Points

• We introduced and evaluated a self-test for Dutch breast screening radiologists.

• ROC curves, case and lesion sensitivity and recall agreement were all satisfactory.

• Agreement in BI-RADS interpretation and description of abnormalities could be improved.

• These are areas that should be targeted with further education and training.

• We offered individual feedback on interpretative performance and overall group feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Holland R, Rijken HJ, Hendriks JH (2007) The Dutch population-based mammography screening: 30-years experience. Breast Care 2:12–18

    Article  Google Scholar 

  2. National Evaluation Team for Breast Cancer Screening (2012) Preliminary results breast cancer screening programme in the Netherlands, 2011. NETCB, Rotterdam

    Google Scholar 

  3. Australian Government Department of Health and Aging (2009) BreastScreen Australia Evaluation: evaluation final report. Department of Health and Aging, Canberra

    Google Scholar 

  4. The Royal College of Radiologists (2007) Standards for self-assessment of performance. RCR, London

    Google Scholar 

  5. United States Food and Drug Administration (2012) Mammography quality standards act and program 2012. http://www.fda.gov/radiationemitting-products/mammographyqualitystandardsactandprogram/default.htm. Accessed 21 Nov 2012

  6. Soh BP, Lee W, Kench PL et al (2012) Assessing reader performance in radiology, an imperfect science: lessons from breast screening. Clin Radiol 67:623–628

    Article  CAS  PubMed  Google Scholar 

  7. Scott HJ, Gale A (2006) Breast screening: PERFORMS identifies key mammographic training needs. Br J Radiol 79:S127–S133

    Article  PubMed  Google Scholar 

  8. Cook AJ, Elmore JG, Zhu W et al (2012) Mammographic interpretation: radiologists' ability to accurately estimate their performance and compare it with that of their peers. AJR Am J Roentgenol 199:695–702

    Article  PubMed Central  PubMed  Google Scholar 

  9. Geller BM, Ichikawa L, Miglioretti DL, Eastman D (2012) Web-based mammography audit feedback. AJR Am J Roentgenol 198:562–567

    Article  Google Scholar 

  10. Shapiro S, Coleman EA, Broeders M et al (1998) Breast cancer screening programmes in 22 countries: current policies, administration and guidelines. International Breast Cancer Screening Network (IBSN) and the European Network of Pilot Projects for Breast Cancer Screening. Int J Epidemiol 27:735–742

    Article  CAS  PubMed  Google Scholar 

  11. Yankaskas BC, Klabunde CN, Ancelle-Park R et al (2004) International comparison of performance measures for screening mammography: can it be done? J Med Screen 11:187–193

    Article  CAS  PubMed  Google Scholar 

  12. Timmers JM, van Doorne-Nagtegaal HJ, Zonderland HM et al (2012) The Breast Imaging Reporting and Data System (BI-RADS) in the Dutch breast cancer screening programme: its role as an assessment and stratification tool. Eur Radiol 22:5

    Article  Google Scholar 

  13. Eng J (2012) ROC analysis: web-based calculater for ROC curves. http://www.jrocfit.org. Accessed 21 Nov 2012

  14. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  PubMed  Google Scholar 

  15. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20:37–46

    Article  Google Scholar 

  16. Ciatto S, Ambrogetti D, Morrone D, Del Turco M (2006) Analysis of the results of a proficiency test in screening mammography at the CSPO of Florence: review of 705 tests. Radiol Med 111:797–803

    Article  CAS  PubMed  Google Scholar 

  17. Onega T, Smith M, Miglioretti DL et al (2012) Radiologist agreement for mammographic recall by case difficulty and finding type. J Am Coll Radiol 9:788–794

    Article  PubMed Central  PubMed  Google Scholar 

  18. Reed WM, Lee WB, Cawson JN, Brennan PC (2010) Malignancy detection in digital mammograms: important reader characteristics and required case numbers. Acad Radiol 17:1409–1413

    Article  PubMed  Google Scholar 

  19. Esserman L, Cowley H, Eberle C et al (2002) Improving the accuracy of mammography: volume and outcome relationships. J Natl Cancer Inst 94(5):369–375

    Article  PubMed  Google Scholar 

  20. Miglioretti DL, Gard CC, Carney PA et al (2009) When radiologists perform best: the learning curve in screening mammogram interpretation. Radiology 253:632–640

    Article  PubMed  Google Scholar 

  21. Pusic MV, Andrews JS, Kessler DO et al (2012) Prevalence of abnormal cases in an image bank affects the learning of radiograph interpretation. Med Educ 46:289–298

    Article  PubMed  Google Scholar 

  22. Carney PA, Bogart TA, Geller BM et al (2012) Association between time spent interpreting, level of confidence, and accuracy of screening mammography. Am J Roentgenol 198:970–978

    Article  Google Scholar 

  23. Elmore JG, Jackson SL, Abraham L et al (2009) Variability in interpretive performance at screening mammography and radiologists' characteristics associated with accuracy. Radiology 253:641–651

    Article  PubMed  Google Scholar 

  24. Kerlikowske K, Grady D, Barclay J et al (1998) Variability and accuracy in mammographic interpretation using the American College of Radiology Breast Imaging Reporting and Data System. J Natl Cancer Inst 90:1801–1809

    Article  CAS  PubMed  Google Scholar 

  25. Lazarus E, Mainiero MB, Schepps B et al (2006) BI-RADS lexicon for US and mammography: interobserver variability and positive predictive value. Radiology 239:385–391

    Article  PubMed  Google Scholar 

  26. Pinto A, Acampora C, Pinto F et al (2011) Learning from diagnostic errors: a good way to improve education in radiology. Eur J Radiol 78:372–376

    Article  PubMed  Google Scholar 

  27. Gur D, Bandos AI, Cohen CS et al (2008) The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 249:47–53

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors especially would like to acknowledge the careful work of Paul van de Looi in organising and developing the test set. We also thank the expert panel for their time in composing the test set and the participating radiologists for providing the data for this work

Author information

Authors and Affiliations

  1. National Expert and Training Centre for Breast Cancer Screening, PO Box 6873, 6503 GJ, Nijmegen, The Netherlands

    J. M. H. Timmers, A. L. M. Verbeek, R. M. Pijnappel, M. J. M. Broeders & G. J. den Heeten

  2. Department for Health Evidence, Radboud University Medical Centre, PO Box 9101, 6500 HB, Nijmegen, The Netherlands

    J. M. H. Timmers, A. L. M. Verbeek & M. J. M. Broeders

  3. Department of Radiology, University Medical Centre Utrecht, PO Box 85500, 3508 GA, Utrecht, The Netherlands

    R. M. Pijnappel

  4. Department of Radiology, Academic Medical Centre, University of Amsterdam, PO Box 22660, 1100 DD, Amsterdam, The Netherlands

    G. J. den Heeten

Authors
  1. J. M. H. Timmers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. L. M. Verbeek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. M. Pijnappel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. J. M. Broeders
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. G. J. den Heeten
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. M. H. Timmers.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Timmers, J.M.H., Verbeek, A.L.M., Pijnappel, R.M. et al. Experiences with a self-test for Dutch breast screening radiologists: lessons learnt. Eur Radiol 24, 294–304 (2014). https://doi.org/10.1007/s00330-013-3018-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-013-3018-4

Keywords

Search

Navigation