Journal of Digital Imaging

, Volume 32, Issue 5, pp 746–760 | Cite as

Can a Machine Learn from Radiologists’ Visual Search Behaviour and Their Interpretation of Mammograms—a Deep-Learning Study

  • Suneeta MallEmail author
  • Patrick C. Brennan
  • Claudia Mello-Thoms


Visual search behaviour and the interpretation of mammograms have been studied for errors in breast cancer detection. We aim to ascertain whether machine-learning models can learn about radiologists’ attentional level and the interpretation of mammograms. We seek to determine whether these models are practical and feasible for use in training and teaching programmes. Eight radiologists of varying experience levels in reading mammograms reviewed 120 two-view digital mammography cases (59 cancers). Their search behaviour and decisions were captured using a head-mounted eye-tracking device and software allowing them to record their decisions. This information from radiologists was used to build an ensembled machine-learning model using top-down hierarchical deep convolution neural network. Separately, a model to determine type of missed cancer (search, perception or decision-making) was also built. Analysis and comparison of variants of these models using different convolution networks with and without transfer learning were also performed. Our ensembled deep-learning network architecture can be trained to learn about radiologists’ attentional level and decisions. High accuracy (95%, p value ≅ 0 [better than dumb/random model]) and high agreement between true and predicted values (kappa = 0.83) in such modelling can be achieved. Transfer learning techniques improve by < 10% with the performance of this model. We also show that spatial convolution neural networks are insufficient in determining the type of missed cancers. Ensembled hierarchical deep convolution machine-learning models are plausible in modelling radiologists’ attentional level and their interpretation of mammograms. However, deep convolution networks fail to characterise the type of false-negative decisions.


Visual search Breast cancer Deep learning Mammography Machine learning 



Craniocaudal view


Mediolateral oblique view


Eye tracking


Visual search map


Foveal area


Peripheral area


Never fixated area


True positives


False positives


True negatives


False negatives


Machine learning models


Deep convolution neural network


Residual network


Neural architecture search network


Visual geometry group network


(Eye) Attentional level and decision


Missed cancer



We would like to thank the radiologists that participated in our experiment.


This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.


  1. 1.
    AIHW: Cancer in Australia 2017," in Cancer series no. 101. Cat. No. CAN 100. Canberra: AIHW, 2017Google Scholar
  2. 2.
    (May, 2018). Australian Institute of Health and Welfare 2017. Australian Cancer Incidence and Mortality (ACIM) books: Breast Cancer. Available:
  3. 3.
    S. I. Ferlay J, Ervik M, Dikshit R, Eser S, Mathers C, Rebelo M, Parkin DM, Forman D, Bray, F. (2014, 16/1/2015). GLOBOCAN 2012 v1.1, Cancer Incidence and Mortality Worldwide: IARC Cancer Base No. 11 [Internet]. Available:
  4. 4.
    S. Mall, S. Lewis, P. Brennan, J. Noakes, and C. Mello-Thoms, "The role of digital breast tomosynthesis in the breast assessment clinic: a review, " Journal of Medical Radiation Sciences, pp. n/a-n/a, 2017.Google Scholar
  5. 5.
    Nelson HD, Fu R, Cantor A, Pappas M, Daeges M, Humphrey L: Effectiveness of breast cancer screening: systematic review and meta-analysis to update the 2009 U.S. preventive services task force recommendation. Annals of Internal Medicine 164(4):244, 2016CrossRefGoogle Scholar
  6. 6.
    Huynh PT, Jarolimek AM, Daye S: The false-negative mammogram. Radiographics 18(5):1137–1154, 1998CrossRefGoogle Scholar
  7. 7.
    Alakhras M, Bourne R, Rickard M, Ng KH, Pietrzyk M, Brennan PC: Digital tomosynthesis: a new future for breast imaging? Clinical Radiology 68(5):e225–e236, 2013–May 2013CrossRefGoogle Scholar
  8. 8.
    Kundel HL, Nodine CF, Toto L: Searching for lung nodules. The guidance of visual scanning," (in eng). Invest Radiol 26(9):777–781, Sep 1991Google Scholar
  9. 9.
    Tuddenham WJ: Visual search, image organization, and reader error in roentgen diagnosis. Radiology 78(5):694–704, 1962CrossRefGoogle Scholar
  10. 10.
    Kundel HL, Lafollet PS: Visual search patterns and experience with radiological images. Radiology 103(3):523, 1972CrossRefGoogle Scholar
  11. 11.
    Mello-Thoms C et al.: Different search patterns and similar decision outcomes: how can experts agree in the decisions they make when reading digital mammograms? In: Krupinski EA Ed.. Lecture Notes in Computer ScienceDigital Mammography, Proceedings, Vol. 5116, 2008, pp. 212–219CrossRefGoogle Scholar
  12. 12.
    Krupinski EA: Visual scanning patterns of radiologists searching mammograms. Academic Radiology 3(2):137–144, Feb 1996CrossRefGoogle Scholar
  13. 13.
    Kundel HL, Nodine CF, Conant EF, Weinstein SP: Holistic component of image perception in mammogram interpretation: gaze-tracking study. Radiology 242(2):396–402, Feb 2007CrossRefGoogle Scholar
  14. 14.
    Kundel HL, Nodine CF, Carmody D: Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. (in eng), Invest Radiol 13(3):175–181, May-Jun 1978CrossRefGoogle Scholar
  15. 15.
    Nodine CF, Kundel HL: Using eye movements to study visual search and to improve tumor detection. Radiographics: a Review Publication of the Radiological Society of North America, Inc. 7(6):1241–1250, 1987–Nov 1987CrossRefGoogle Scholar
  16. 16.
    Kundel HL, Nodine CF, Krupinski EA: Searching for lung nodules—visual dwell indicates locations of false-positive and false-negative decisions. Investigative Radiology 24(6):472–478, Jun 1989CrossRefGoogle Scholar
  17. 17.
    Mello-Thoms C, Dunn S, Nodine CF, Kundel HL: Image structure and perceptual errors in mammogram reading: a pilot study. In: Krupinski EA Ed.. (Proceedings of the Society of Photo-Optical Instrumentation Engineers (Spie), no. 26)Medical Imaging 2000: Image Perception and Performance, Vol. 1, 2000, pp. 170–173Google Scholar
  18. 18.
    Nodine CF, Mello-Thoms C, Weinstein SP, Kundel HL, Toto LC: Do subtle breast cancers attract visual attention during initial impression? In: Krupinski EA Ed.. Ed. (Proceedings of the Society of Photo-Optical Instrumentation Engineers (Spie), no. 26)Medical Imaging 2000: Image Perception and Performance, Vol. 1, 2000, pp. 156–159CrossRefGoogle Scholar
  19. 19.
    Mall S, Brennan P, Mello-Thoms C: Fixated and not fixated regions of mammograms: a higher-order statistical analysis of visual search behavior. Academic Radiology 24(4):442–455, 2017CrossRefGoogle Scholar
  20. 20.
    Mello-Thoms C, Dunn S, Nodine CF, Kundel HL, Weinstein SP: The perception of breast cancer: what differentiates missed from reported cancers in mammography? Academic Radiology 9(9):1004–1012, Sep 2002CrossRefGoogle Scholar
  21. 21.
    Mello-Thoms C, Dunn SM, Nodine CF, Kundel HL: The perception of breast cancers—a spatial frequency analysis of what differentiates missed from reported cancers. Ieee Transactions on Medical Imaging 22(10):1297–1306, Oct 2003CrossRefGoogle Scholar
  22. 22.
    Mello-Thoms C, Nodine CF, Kundel HL: Relating image based features to mammogram interpretation. In: Medical Imaging 2002 Conference, San Diego, CA, 2002, Vol. e4686, 2002, pp. 80–83Google Scholar
  23. 23.
    Berbaum KS et al.: The influence of clinical history on visual-search with single and multiple abnormalities. Investigative Radiology 28(3):191–201, Mar 1993CrossRefGoogle Scholar
  24. 24.
    Samei E, Krupinski EA: The Handbook of Medical Image Perception and Techniques (no. Book, Whole). Cambridge: Cambridge University Press, 2010Google Scholar
  25. 25.
    Mall S, Brennan PC, Mello-Thoms C: A deep (learning) dive into visual search behaviour of breast radiologists. SPIE Medical Imaging 10577:11, 2018 SPIEGoogle Scholar
  26. 26.
    Hillstrom AP: Repetition effects in visual search," (in eng). Percept Psychophys 62(4):800–817, May 2000Google Scholar
  27. 27.
    Kok EM, Jarodzka H, de Bruin ABH, BinAmir HAN, Robben SGF, van Merriënboer JJG: Systematic viewing in radiology: seeing more, missing less? Advances in Health Sciences Education 21:189–205, 07/16 2016CrossRefGoogle Scholar
  28. 28.
    D. M Mount, S. Arya, S. E. Kemp, and G. Jefferis. (2015). Fast Nearest Neighbour Search (Wraps Arya and Mount's ANN: A Library for Approximate Nearest Neighbor Searching). Available: and
  29. 29.
    X. Z. Kaiming He, Shaoqing Ren, Jian Sun, "Deep Residual Learning for Image Recognition," vol. arXiv:1512.03385, no., 2015.Google Scholar
  30. 30.
    C. Szegedy, Ioffe, S., Vanhoucke, V., "Inception-v4, Inception-resnet and the Impact of Residual Connections on Learning," vol. arXiv:1602.07261, no., 2016.Google Scholar
  31. 31.
    V. V. Barret Zoph, Jonathon Shlens, Quoc V. Le, "Learning Transferable Architectures for Scalable Image Recognition," vol. arXiv:1707.07012, no.
  32. 32.
    A. Z. Karen Simonyan, "Very Deep Convolutional Networks for Large-Scale Image Recognition," vol. arXiv:1409.1556, no., 2014.Google Scholar
  33. 33.
    Arel I, Rose DC, Karnowski TP: Research frontier: deep machine learning—a new frontier in artificial intelligence research. Comp. Intell. Mag. 5(4):13–18, 2010CrossRefGoogle Scholar
  34. 34.
    Greenspan H, Ginneken BV, Summers RM: Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging 35(5):1153–1159, 2016CrossRefGoogle Scholar
  35. 35.
    Pan SJ, Yang Q: A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10):1345–1359, 2010CrossRefGoogle Scholar
  36. 36.
    C. N. Silla and A. A. Freitas, "A survey of hierarchical classification across different application domains," Data Mining and Knowledge Discovery, journal article vol. 22, no. 1, pp. 31–72, January 01 2011.Google Scholar
  37. 37.
    A. M. Mateusz Buda, Maciej A. Mazurowski, "A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks," vol., no. arXiv:1710.05381, 2017.
  38. 38.
    Tsehay YK et al.: Convolutional neural network based deep-learning architecture for prostate cancer detection on multiparametric magnetic resonance images. SPIE Medical Imaging 10134:11, 2017 SPIEGoogle Scholar
  39. 39.
    Sokolova M, Lapalme G: A systematic analysis of performance measures for classification tasks. Information Processing & Management 45(4):427–437, 2009/07/01/, 2009CrossRefGoogle Scholar
  40. 40.
    Manning DJ, Ethell SC, Donovan T: Detection or decision errors? Missed lung cancer from the posteroanterior chest radiograph. The British Journal of Radiology 77(915):231–235, 2004CrossRefGoogle Scholar
  41. 41.
    Donovan T, Manning DJ: Successful reporting by non-medical practitioners such as radiographers, will always be task-specific and limited in scope. Radiography 12(1):7–12, 2006/02/01/, 2006CrossRefGoogle Scholar
  42. 42.
    Litchfield D, Ball LJ, Donovan T, Manning DJ, Crawford T: Viewing another person's eye movements improves identification of pulmonary nodules in chest X-ray inspection. Journal of Experimental Psychology: Applied 16(3):251–262, 2010Google Scholar
  43. 43.
    Mello-Thoms C: Perception of breast cancer: eye-position analysis of mammogram interpretation. Academic Radiology 10(1):4–12, Jan 2003CrossRefGoogle Scholar
  44. 44.
    Gandomkar Z, Tay K, Brennan PC, Mello-Thoms C: A Model Based on Temporal Dynamics of Fixations for Distinguishing Expert Radiologists' Scanpaths, Vol. 10136, 2017, pp. 1013606-1013606-9Google Scholar
  45. 45.
    A. R. Z. Ashesh Jain, Silvio Savarese, Ashutosh Saxena, "Structural-RNN: Deep Learning on Spatio-Temporal Graphs," , vol. arXiv:1511.05298, 2016.Google Scholar
  46. 46.
    H. Y. Bing Yu, Zhanxing Zhu, "Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting," , vol. arXiv:1709.04875, 2018.Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  1. 1.Medical Image Optimisation and Perception Research Group (MIOPeG), Faculty of Medicine and HealthUniversity of SydneyLidcombeAustralia
  2. 2.Department of RadiologyUniversity of IowaIowa CityUSA

Personalised recommendations