Multi-task SonoEyeNet: Detection of Fetal Standardized Planes Assisted by Generated Sonographer Attention Maps

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11070)


We present a novel multi-task convolutional neural network called Multi-task SonoEyeNet (M-SEN) that learns to generate clinically relevant visual attention maps using sonographer gaze tracking data on input ultrasound (US) video frames so as to assist standardized abdominal circumference (AC) plane detection. Our architecture consists of a generator and a discriminator, which are trained in an adversarial scheme. The generator learns sonographer attention on a given US video frame to predict the frame label (standardized AC plane/background). The discriminator further fine-tunes the predicted attention map by encouraging it to mimick the ground-truth sonographer attention map. The novel model expands the potential clinical usefulness of a previous model by eliminating the requirement of input gaze tracking data during inference without compromising its plane detection performance (Precision: 96.8, Recall: 96.2, F-1 score: 96.5).


Multi-task learning Generative Adversarial Network Gaze tracking Fetal ultrasound Saliency prediction Standardized plane detection 



We acknowledge the ERC (ERC-ADG-2015 694581 for project PULSE) and the EPSRC (EP/GO36861/1, and EP/MO13774/1).


  1. 1.
    Hack, M., et al.: Outcomes of extremely low birth weight infants. Pediatrics 98(5), 931–937 (1996)Google Scholar
  2. 2.
    Sarris, I., et al.: Intra- and inter-observer variability in fetal ultrasound measurements. Ultrasound Obstet. Gynecol. 39(3), 266–273 (2012)CrossRefGoogle Scholar
  3. 3.
    Yaqub, M., Kelly, B., Papageorghiou, A.T., Noble, J.A.: Guided random forests for identification of key fetal anatomy and image categorization in ultrasound scans. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 687–694. Springer, Cham (2015). Scholar
  4. 4.
    Baumgartner, C.F., et al.: SonoNet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound. IEEE Trans. Med. Imaging 36(11), 2204–2215 (2017)CrossRefGoogle Scholar
  5. 5.
    Cai, Y., et al.: SonoEyeNet: standardized fetal ultrasound plane detection informed by eye tracking. In: IEEE ISBI, pp. 1475–1478 (2018)Google Scholar
  6. 6.
    Koch, C., Ullman, S.: Shifts in selective visual attention: towards the underlying neural circuitry. In: Vaina, L.M. (ed.) Matters of Intelligence. Synthese Library (Studies in Epistemology, Logic, Methodology, and Philosophy of Science), vol. 188, pp. 115–141. Springer, Dordrecht (1987). Scholar
  7. 7.
    Kümmerer, M., et al.: DeepGaze II: Reading fixations from deep features trained on object recognition. arXiv preprint arXiv:1610.01563 (2016)
  8. 8.
    Pan, J., et al.: SalGAN: Visual Saliency Prediction with Generative Adversarial Networks. In: arXiv:1701.0181v2 [cs.CV] (2017)
  9. 9.
    Ahmed, M., et al.: An eye-tracking inspired method for standardised plane extraction from fetal abdominal ultrasound volumes. In: ISBI, pp. 1084–1087. IEEE (2016)Google Scholar
  10. 10.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014)Google Scholar
  11. 11.
    Kruthiventi, S.S., et al.: Deepfix: a fully convolutional neural network for predicting human eye fixations. IEEE Trans. Image Process. 26(9), 4446–4456 (2017)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Huang, X., et al.: SALICON: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In: ICCV, pp. 262–270 (2015)Google Scholar
  13. 13.
    Radford, A., et al.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  14. 14.
    Fuchs, A.F.: The saccadic system. In: The Control of Eye Movements, pp. 343–362 (1971)Google Scholar
  15. 15.
    Mathe, S., Sminchisescu, C.: Dynamic eye movement datasets and learnt saliency models for visual action recognition. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, pp. 842–856. Springer, Heidelberg (2012). Scholar
  16. 16.
    Bylinskii, Z., et al.: What do different evaluation metrics tell us about saliency models? arXiv preprint arXiv:1604.03605 (2016)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Institute of Biomedical EngineeringUniversity of OxfordOxfordUK

Personalised recommendations