Abstract
We investigate the potential contribution of an AI system as a safety net application for radiologists in breast cancer screening. As a safety net, the AI alerts on cases suspected to be malignant which the radiologist did not recommend for a recall. We analyzed held-out data of 2,638 exams enriched with 90 missed cancers. In screening mammography settings, we show that a system alerting on 11 out of every 1,000 cases, could detect up to 10.7% of the radiologists’ missed cancers. Thus, significantly increasing radiologist’s sensitivity to 80.3%, while only slightly decreasing their specificity to 95.3%. Importantly, the safety net demonstrated a significant contribution to their performance even when radiologists utilized both mammography and ultrasound images. In those settings, it would have alerted 8.5 times per 1,000 cases, and detected 11.7% of the radiologists’ missed cancers. In an analysis of the missed cancers by an expert, we found that most of the cancers detected by the AI were visible post-hoc. Finally, we performed a reader study with five radiologists over 120 exams, 10 of which were originally missed cancers. The AI safety net was able to assist 3 out of the 5 radiologists in detecting missed cancers without raising any false alerts.
Keywords
- Computer-aided diagnosis
- Deep learning
- Breast imaging
This is a preview of subscription content, access via your institution.
Buying options
Change history
29 September 2020
The original version of this chapter was revised. Dr. Ayelet Akselrod-Ballin contributed to the development of the conference paper and was therefore added to the list of coauthors.
References
Lehman, C.D., et al.: National performance benchmarks for modern screening digital mammography: update from the Breast Cancer Surveillance Consortium. Radiology 283, 49–58 (2016)
Antonio, A.L.M., Crespi, C.M.: Predictors of interobserver agreement in breast imaging using the Breast Imaging Reporting and Data System. Breast Cancer Res. Treat. 120, 539–546 (2010). https://doi.org/10.1007/s10549-010-0770-x
Nishikawa, R.M., Comstock, C.E., Linver, M.N., Newstead, G.M., Sandhir, V., Schmidt, R.A.: Agreement between radiologists’ interpretations of screening mammograms. In: Tingberg, A., Lång, K., Timberg, Pontus (eds.) IWDM 2016. LNCS, vol. 9699, pp. 3–10. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41546-8_1
Katalinic, A., Bartel, C., Raspe, H., Schreer, I.: Beyond mammography screening: quality assurance in breast cancer diagnosis (The QuaMaDi Project). Br. J. Cancer 96, 157 (2007)
Karssemeijer, N.: Effect of independent double and multiple reading of screening mammograms by breast density. 1136 words (2014). https://doi.org/10.1594/ecr2014/c-0358
Taylor-Phillips, S., Jenkinson, D., Stinton, C., Wallis, M.G., Dunn, J., Clarke, A.: Double reading in breast cancer screening: cohort evaluation in the CO-OPS trial. Radiology 287, 749–757 (2018). https://doi.org/10.1148/radiol.2018171010
Leivo, T., et al.: Incremental cost-effectiveness of double-reading mammograms. Breast Cancer Res. Treat. 54, 261–267 (1999). https://doi.org/10.1023/A:1006136107092
Lehman, C.D., Wellman, R.D., Buist, D.S.M., Kerlikowske, K., Tosteson, A.N.A., Miglioretti, D.L.: Breast cancer surveillance consortium: diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern. Med. 175, 1828–1837 (2015). https://doi.org/10.1001/jamainternmed.2015.5231
Cole, E.B., Zhang, Z., Marques, H.S., Edward Hendrick, R., Yaffe, M.J., Pisano, E.D.: Impact of computer-aided detection systems on radiologist accuracy with digital mammography. Am. J. Roentgenol. 203, 909–916 (2014). https://doi.org/10.2214/AJR.12.10187
Gao, Y., Geras, K.J., Lewin, A.A., Moy, L.: New frontiers: an update on computer-aided diagnosis for breast imaging in the age of artificial intelligence. AJR Am. J. Roentgenol. 212, 300–307 (2019). https://doi.org/10.2214/AJR.18.20392
Akselrod-Ballin, A., et al.: Predicting breast cancer by applying deep learning to linked health records and mammograms. Radiology 292, 331–342 (2019). https://doi.org/10.1148/radiol.2019182622
Yala, A., Lehman, C., Schuster, T., Portnoi, T., Barzilay, R.: A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 292, 60–66 (2019). https://doi.org/10.1148/radiol.2019182716
McKinney, S.M., et al.: International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020). https://doi.org/10.1038/s41586-019-1799-6
Kim, H.-E., et al.: Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digit. Health 2, e138–e148 (2020). https://doi.org/10.1016/S2589-7500(20)30003-0
Schaffter, T., et al.: Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms. JAMA Netw. Open 3, e200265 (2020). https://doi.org/10.1001/jamanetworkopen.2020.0265
Rodriguez-Ruiz, A., et al.: Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists. J. Natl Cancer Inst. 111, 916–922 (2019). https://doi.org/10.1093/jnci/djy222
Siu, A.L.: Screening for breast cancer: U.S. preventive services task force recommendation statement. Ann. Intern. Med. 164, 279 (2016). https://doi.org/10.7326/M15-2886. On behalf of the U.S. Preventive Services Task Force
Alcusky, M., Philpotts, L., Bonafede, M., Clarke, J., Skoufalos, A.: The patient burden of screening mammography recall. J. Womens Health 23, S-11 (2014). https://doi.org/10.1089/jwh.2014.1511
Funding
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 813533.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chorev, M. et al. (2020). The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net. In: , et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12266. Springer, Cham. https://doi.org/10.1007/978-3-030-59725-2_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-59725-2_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59724-5
Online ISBN: 978-3-030-59725-2
eBook Packages: Computer ScienceComputer Science (R0)
-
Published in cooperation with
http://miccai.org/
