Journal of Digital Imaging

, Volume 30, Issue 2, pp 234–243 | Cite as

Transfer Learning with Convolutional Neural Networks for Classification of Abdominal Ultrasound Images

  • Phillip M. ChengEmail author
  • Harshawn S. Malhi


The purpose of this study is to evaluate transfer learning with deep convolutional neural networks for the classification of abdominal ultrasound images. Grayscale images from 185 consecutive clinical abdominal ultrasound studies were categorized into 11 categories based on the text annotation specified by the technologist for the image. Cropped images were rescaled to 256 × 256 resolution and randomized, with 4094 images from 136 studies constituting the training set, and 1423 images from 49 studies constituting the test set. The fully connected layers of two convolutional neural networks based on CaffeNet and VGGNet, previously trained on the 2012 Large Scale Visual Recognition Challenge data set, were retrained on the training set. Weights in the convolutional layers of each network were frozen to serve as fixed feature extractors. Accuracy on the test set was evaluated for each network. A radiologist experienced in abdominal ultrasound also independently classified the images in the test set into the same 11 categories. The CaffeNet network classified 77.3% of the test set images accurately (1100/1423 images), with a top-2 accuracy of 90.4% (1287/1423 images). The larger VGGNet network classified 77.9% of the test set accurately (1109/1423 images), with a top-2 accuracy of VGGNet was 89.7% (1276/1423 images). The radiologist classified 71.7% of the test set images correctly (1020/1423 images). The differences in classification accuracies between both neural networks and the radiologist were statistically significant (p < 0.001). The results demonstrate that transfer learning with convolutional neural networks may be used to construct effective classifiers for abdominal ultrasound images.


Machine learning Classification Artificial neural networks Digital image processing Deep learning 


  1. 1.
    LeCun Y, Bengio Y, Hinton G: Deep learning. Nature 521:436–444, 2015CrossRefPubMedGoogle Scholar
  2. 2.
    Goodfellow I, Bengio Y, Courville A: Deep Learning, MIT Press (in preparation), 2016Google Scholar
  3. 3.
    Thrall JH: Trends and Developments Shaping the Future of Diagnostic Medical Imaging: 2015 Annual Oration in Diagnostic Radiology. Radiology 279:660–666, 2016CrossRefPubMedGoogle Scholar
  4. 4.
    Greenspan H, van Ginneken B, Summers RM: Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique. IEEE Trans Med Imaging 35:1153–1159, 2016CrossRefGoogle Scholar
  5. 5.
    Kato H, Kanematsu M, Zhang X, Saio M, Kondo H, Goshima S, Fujita H: Computer-Aided Diagnosis of Hepatic Fibrosis: Preliminary Evaluation of MRI Texture Analysis Using the Finite Difference Method and an Artificial Neural Network. Am J Roentgenol 189:117–122, 2007CrossRefGoogle Scholar
  6. 6.
    Ayer T, Chhatwal J, Alagoz O, Kahn CE, Woods RW, Burnside ES: Comparison of Logistic Regression and Artificial Neural Network Models in Breast Cancer Risk Estimation. RadioGraphics 30:13–22, 2010CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Preis O, Blake MA, Scott JA: Neural Network Evaluation of PET Scans of the Liver: A Potentially Useful Adjunct in Clinical Interpretation. Radiology 258:714–721, 2011CrossRefPubMedGoogle Scholar
  8. 8.
    Krizhevsky A, Sutskever I, Hinton GE: ImageNet Classification with Deep Convolutional Neural Networks. In: Advances in Neural Information Processing Systems (NIPS 2012). Lake Tahoe, 2012Google Scholar
  9. 9.
    Simonyan K, Zisserman A: Very Deep Convolutional Networks for Large-Scale Image Recognition. In: International Conference on Learning Representations 2015. San Diego, 2014Google Scholar
  10. 10.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, 2015, pp 1–9Google Scholar
  11. 11.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L: ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis 115:211–252, 2015CrossRefGoogle Scholar
  12. 12.
    Cho J, Lee K, Shin E, Choy G, Do S: How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv:1511.06348, 2015Google Scholar
  13. 13.
    Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J: Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Trans Med Imaging 35:1299–1312, 2016CrossRefPubMedGoogle Scholar
  14. 14.
    Shin HC, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM: Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE Trans Med Imaging 35:1285–1298, 2016CrossRefPubMedGoogle Scholar
  15. 15.
    Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S: Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network. IEEE Trans Med Imaging 35:1207–1216, 2016CrossRefPubMedGoogle Scholar
  16. 16.
    Rajkomar A, Lingam S, Taylor AG, Blum M, Mongan J: High-Throughput Classification of Radiographs Using Deep Convolutional Neural Networks. J Digit Imaging, 2016Google Scholar
  17. 17.
    Razavian AS, Azizpour H, Sullivan J, Carlsson S: CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Columbus, 2014, pp 512–519Google Scholar
  18. 18.
    Karpathy A: CS231n Course Notes: Transfer Learning. [Online]. Available: [Accessed: 19-May-2016]
  19. 19.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T: Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv:1408.5093, 2014Google Scholar
  20. 20.
    Donahue J: CaffeNet (GitHub Page). [Online]. Available: [Accessed: 16-May-2016]
  21. 21.
    Simonyan K: VGG team ILSVRC-2014 model with 16 weight layers (GitHub Page). [Online]. Available: [Accessed: 16-May-2016]
  22. 22.
    van der Maaten L, Hinton G: Visualizing Data using t-SNE. J Mach Learn Res 9:2579–2605, 2008Google Scholar
  23. 23.
    Pedregosa F, Varoquaux G, Gramfort A, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D: Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830, 2011Google Scholar
  24. 24.
    R Core Team: R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, 2016Google Scholar
  25. 25.
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R: Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J Mach Learn Res 15:1929–1958, 2014Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2016

Authors and Affiliations

  1. 1.Department of RadiologyKeck School of Medicine of USCLos AngelesUSA
  2. 2.USC Norris Cancer Center and HospitalLos AngelesUSA

Personalised recommendations