Can a Machine Learn from Radiologists’ Visual Search Behaviour and Their Interpretation of Mammograms—a Deep-Learning Study
Visual search behaviour and the interpretation of mammograms have been studied for errors in breast cancer detection. We aim to ascertain whether machine-learning models can learn about radiologists’ attentional level and the interpretation of mammograms. We seek to determine whether these models are practical and feasible for use in training and teaching programmes. Eight radiologists of varying experience levels in reading mammograms reviewed 120 two-view digital mammography cases (59 cancers). Their search behaviour and decisions were captured using a head-mounted eye-tracking device and software allowing them to record their decisions. This information from radiologists was used to build an ensembled machine-learning model using top-down hierarchical deep convolution neural network. Separately, a model to determine type of missed cancer (search, perception or decision-making) was also built. Analysis and comparison of variants of these models using different convolution networks with and without transfer learning were also performed. Our ensembled deep-learning network architecture can be trained to learn about radiologists’ attentional level and decisions. High accuracy (95%, p value ≅ 0 [better than dumb/random model]) and high agreement between true and predicted values (kappa = 0.83) in such modelling can be achieved. Transfer learning techniques improve by < 10% with the performance of this model. We also show that spatial convolution neural networks are insufficient in determining the type of missed cancers. Ensembled hierarchical deep convolution machine-learning models are plausible in modelling radiologists’ attentional level and their interpretation of mammograms. However, deep convolution networks fail to characterise the type of false-negative decisions.
KeywordsVisual search Breast cancer Deep learning Mammography Machine learning
Mediolateral oblique view
Visual search map
Never fixated area
Machine learning models
Deep convolution neural network
Neural architecture search network
Visual geometry group network
(Eye) Attentional level and decision
We would like to thank the radiologists that participated in our experiment.
This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.
- 1.AIHW: Cancer in Australia 2017," in Cancer series no. 101. Cat. No. CAN 100. Canberra: AIHW, 2017Google Scholar
- 2.(May, 2018). Australian Institute of Health and Welfare 2017. Australian Cancer Incidence and Mortality (ACIM) books: Breast Cancer. Available: https://www.aihw.gov.au/reports/cancer/acim-books
- 3.S. I. Ferlay J, Ervik M, Dikshit R, Eser S, Mathers C, Rebelo M, Parkin DM, Forman D, Bray, F. (2014, 16/1/2015). GLOBOCAN 2012 v1.1, Cancer Incidence and Mortality Worldwide: IARC Cancer Base No. 11 [Internet]. Available: http://globocan.iarc.fr
- 4.S. Mall, S. Lewis, P. Brennan, J. Noakes, and C. Mello-Thoms, "The role of digital breast tomosynthesis in the breast assessment clinic: a review, " Journal of Medical Radiation Sciences, pp. n/a-n/a, 2017.Google Scholar
- 8.Kundel HL, Nodine CF, Toto L: Searching for lung nodules. The guidance of visual scanning," (in eng). Invest Radiol 26(9):777–781, Sep 1991Google Scholar
- 11.Mello-Thoms C et al.: Different search patterns and similar decision outcomes: how can experts agree in the decisions they make when reading digital mammograms? In: Krupinski EA Ed.. Lecture Notes in Computer ScienceDigital Mammography, Proceedings, Vol. 5116, 2008, pp. 212–219CrossRefGoogle Scholar
- 17.Mello-Thoms C, Dunn S, Nodine CF, Kundel HL: Image structure and perceptual errors in mammogram reading: a pilot study. In: Krupinski EA Ed.. (Proceedings of the Society of Photo-Optical Instrumentation Engineers (Spie), no. 26)Medical Imaging 2000: Image Perception and Performance, Vol. 1, 2000, pp. 170–173Google Scholar
- 18.Nodine CF, Mello-Thoms C, Weinstein SP, Kundel HL, Toto LC: Do subtle breast cancers attract visual attention during initial impression? In: Krupinski EA Ed.. Ed. (Proceedings of the Society of Photo-Optical Instrumentation Engineers (Spie), no. 26)Medical Imaging 2000: Image Perception and Performance, Vol. 1, 2000, pp. 156–159CrossRefGoogle Scholar
- 22.Mello-Thoms C, Nodine CF, Kundel HL: Relating image based features to mammogram interpretation. In: Medical Imaging 2002 Conference, San Diego, CA, 2002, Vol. e4686, 2002, pp. 80–83Google Scholar
- 24.Samei E, Krupinski EA: The Handbook of Medical Image Perception and Techniques (no. Book, Whole). Cambridge: Cambridge University Press, 2010Google Scholar
- 25.Mall S, Brennan PC, Mello-Thoms C: A deep (learning) dive into visual search behaviour of breast radiologists. SPIE Medical Imaging 10577:11, 2018 SPIEGoogle Scholar
- 26.Hillstrom AP: Repetition effects in visual search," (in eng). Percept Psychophys 62(4):800–817, May 2000Google Scholar
- 28.D. M Mount, S. Arya, S. E. Kemp, and G. Jefferis. (2015). Fast Nearest Neighbour Search (Wraps Arya and Mount's ANN: A Library for Approximate Nearest Neighbor Searching). Available: https://cran.r-project.org/web/packages/RANN/RANN.pdf and https://www.cs.umd.edu/~mount/ANN/
- 31.V. V. Barret Zoph, Jonathon Shlens, Quoc V. Le, "Learning Transferable Architectures for Scalable Image Recognition," vol. arXiv:1707.07012, no. https://arxiv.org/pdf/1707.07012.pdf.
- 36.C. N. Silla and A. A. Freitas, "A survey of hierarchical classification across different application domains," Data Mining and Knowledge Discovery, journal article vol. 22, no. 1, pp. 31–72, January 01 2011.Google Scholar
- 37.A. M. Mateusz Buda, Maciej A. Mazurowski, "A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks," vol. https://arxiv.org/abs/1710.05381, no. arXiv:1710.05381, 2017.
- 38.Tsehay YK et al.: Convolutional neural network based deep-learning architecture for prostate cancer detection on multiparametric magnetic resonance images. SPIE Medical Imaging 10134:11, 2017 SPIEGoogle Scholar
- 42.Litchfield D, Ball LJ, Donovan T, Manning DJ, Crawford T: Viewing another person's eye movements improves identification of pulmonary nodules in chest X-ray inspection. Journal of Experimental Psychology: Applied 16(3):251–262, 2010Google Scholar
- 44.Gandomkar Z, Tay K, Brennan PC, Mello-Thoms C: A Model Based on Temporal Dynamics of Fixations for Distinguishing Expert Radiologists' Scanpaths, Vol. 10136, 2017, pp. 1013606-1013606-9Google Scholar