Journal of Digital Imaging

, Volume 32, Issue 2, pp 290–299 | Cite as

Integrating Active Learning and Transfer Learning for Carotid Intima-Media Thickness Video Interpretation

  • Zongwei Zhou
  • Jae Shin
  • Ruibin Feng
  • R. Todd Hurst
  • Christopher B. Kendall
  • Jianming LiangEmail author


Cardiovascular disease (CVD) is the number one killer in the USA, yet it is largely preventable (World Health Organization 2011). To prevent CVD, carotid intima-media thickness (CIMT) imaging, a noninvasive ultrasonography method, has proven to be clinically valuable in identifying at-risk persons before adverse events. Researchers are developing systems to automate CIMT video interpretation based on deep learning, but such efforts are impeded by the lack of large annotated CIMT video datasets. CIMT video annotation is not only tedious, laborious, and time consuming, but also demanding of costly, specialty-oriented knowledge and skills, which are not easily accessible. To dramatically reduce the cost of CIMT video annotation, this paper makes three main contributions. Our first contribution is a new concept, called Annotation Unit (AU), which simplifies the entire CIMT video annotation process down to six simple mouse clicks. Our second contribution is a new algorithm, called AFT (active fine-tuning), which naturally integrates active learning and transfer learning (fine-tuning) into a single framework. AFT starts directly with a pre-trained convolutional neural network (CNN), focuses on selecting the most informative and representative AU s from the unannotated pool for annotation, and then fine-tunes the CNN by incorporating newly annotated AU s in each iteration to enhance the CNN’s performance gradually. Our third contribution is a systematic evaluation, which shows that, in comparison with the state-of-the-art method (Tajbakhsh et al., IEEE Trans Med Imaging 35(5):1299–1312, 2016), our method can cut the annotation cost by >81% relative to their training from scratch and >50% relative to their random selection. This performance is attributed to the several advantages derived from the advanced active, continuous learning capability of our AFT method.


Active learning Transfer learning Cardiovascular disease 



This research has been supported partially by NIH under Award Number R01HL128785 and partially by ASU and Mayo Clinic through the Discovery Translation Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.


  1. 1.
    World Health Organization. Global atlas on cardiovascular disease prevention and control. Available at (September 19, 2011)
  2. 2.
    Al Rahhal Ms, Bazi Y, AlHichri H, Alajlan N, Melgani F, Yager R: Deep learning approach for active classification of electrocardiogram signals. Inf Sci 345: 340–354, 2016CrossRefGoogle Scholar
  3. 3.
    Carneiro G, Nascimento J, Bradley A: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: (Navab N, Hornegger J, Wells WM, Frangi AF, Eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, vol. 9351, pp. 652–660. Springer International Publishing, 2015.
  4. 4.
    Chen H, Ni D, Qin J, Li S, Yang X, Wang T, Heng PA: Standard plane localization in fetal ultrasound via domain transferred deep neural networks. IEEE J Biomed Health Inform 19 (5): 1627–1636, 2015CrossRefGoogle Scholar
  5. 5.
    Delsanto S, Molinari F, Giustetto P, Liboni W, Badalamenti S, Suri JS: Characterization of a completely user-independent algorithm for carotid artery segmentation in 2-d ultrasound images. IEEE Trans Instrum Meas 56 (4): 1265–1274, 2007CrossRefGoogle Scholar
  6. 6.
    Deng J, Dong W, Socher R, Li LJ, Li K: Fei-fei, L.: ImageNet: A Large-Scale Hierarchical Image Database CVPR09, 2009Google Scholar
  7. 7.
    Gao M, Bagci U, Lu L, Wu A, Buty M, Shin HC, Roth H, Papadakis GZ, Depeursinge A, Summers RM, et al: Holistic classification of ct attenuation patterns for interstitial lung diseases via deep convolutional neural networks.. In: The 1st workshop on deep learning in medical image analysis, international conference on medical image computing and computer assisted intervention, at MICCAI-DLMIA’15, 2015Google Scholar
  8. 8.
    Gepner AD, Young R, Delaney JA, Tattersall MC, Blaha MJ, Post WS, Gottesman RF, Kronmal R, Budoff MJ, Burke GL, et al: A comparison of coronary artery calcium presence, carotid plaque presence, and carotid intima-media thickness for cardiovascular disease prediction in the multi-ethnic study of atherosclerosis (mesa). Circulation Cardiovascular Imaging 8 (1): e002262, 2015CrossRefGoogle Scholar
  9. 9.
    Greenspan H, van Ginneken B, Summers RM: Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35 (5): 1153–1159, 2016CrossRefGoogle Scholar
  10. 10.
    Guyon I, Cawley G, Dror G, Lemaire V, Statnikov A (2011) JMLR Workshop and conference proceedings (volume 16): Active learning challenge microtome publishingGoogle Scholar
  11. 11.
    He K, Zhang X, Ren S, Sun J: Deep residual learning for image recognition.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016Google Scholar
  12. 12.
    Hoo-Chang S, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM: Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35 (5): 1285, 2016CrossRefGoogle Scholar
  13. 13.
    Hurst RT, Burke RF, Wissner E, Roberts A, Kendall CB, Lester SJ, Somers V, Goldman ME, Wu Q, Khandheria B: Incidence of subclinical atherosclerosis as a marker of cardiovascular risk in retired professional football players. Am J Cardiol 105 (8): 1107–1111, 2010CrossRefGoogle Scholar
  14. 14.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. arXiv:
  15. 15.
    Kass M, Witkin A, Terzopoulos D: Snakes: Active contour models. Int J Comput Vis 1 (4): 321–331, 1988CrossRefGoogle Scholar
  16. 16.
    Krizhevsky A, Sutskever I, Hinton GE: Imagenet classification with deep convolutional neural networks.. In: Advances in neural information processing systems, pp 1097–1105, 2012Google Scholar
  17. 17.
    Li J: Active learning for hyperspectral image classification with a stacked autoencoders based neural network.. In: 2016 IEEE International conference on image processing (ICIP), pp 1062–1065, 2016.
  18. 18.
    Liang J, McInerney T, Terzopoulos D: United snakes. Med Image Anal 10 (2): 215–233, 2006CrossRefGoogle Scholar
  19. 19.
    Liang Q, Wendelhag I, Wikstrand J, Gustavsson T: A multiscale dynamic programming procedure for boundary detection in ultrasonic artery images. IEEE Trans Med Imaging 19 (2): 127–142, 2000CrossRefGoogle Scholar
  20. 20.
    Loizou CP, Pattichis CS, Pantziaris M, Nicolaides A: An integrated system for the segmentation of atherosclerotic carotid plaque. IEEE Trans Inf Technol Biomed 11 (6): 661–667, 2007CrossRefGoogle Scholar
  21. 21.
    Long J, Shelhamer E, Darrell T: Fully convolutional networks for semantic segmentation.. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440, 2015Google Scholar
  22. 22.
    Lorenz MW, Markus HS, Bots ML, Rosvall M, Sitzer M: Prediction of clinical cardiovascular events with carotid intima-media thickness. Circulation 115 (4): 459–467, 2007CrossRefGoogle Scholar
  23. 23.
    Margeta J, Criminisi A, Cabrera Lozoya R, Lee DC, Ayache N: Fine-tuned convolutional neural nets for cardiac mri acquisition plane recognition. Comput Methods Biomech Biomed Eng: Imaging Visual 5 (5): 339–349, 2017Google Scholar
  24. 24.
    Menchón-Lara RM, Bastida-Jumilla MC, González-López A, Sancho-Gómez JL: Automatic evaluation of carotid intima-media thickness in ultrasounds using machine learning.. In: Natural and artificial computation in engineering and medical applications, pp 241–249. Springer, 2013Google Scholar
  25. 25.
    Menchón-Lara RM, Sancho-Gómez JL: Fully automatic segmentation of ultrasound common carotid artery images based on machine learning. Neurocomputing 151: 161–167, 2015CrossRefGoogle Scholar
  26. 26.
    Schlegl T, Ofner J, Langs G: Unsupervised pre-training across image domains improves lung tissue classification.. In: Medical computer vision: Algorithms for big data, pp 82–93. Springer, 2014Google Scholar
  27. 27.
    Shin HC, Lu L, Kim L, Seff A, Yao J, Summers RM: Interleaved text/image deep mining on a very large-scale radiology database.. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1090–1099, 2015Google Scholar
  28. 28.
    Shin J, Tajbakhsh N, Todd Hurst R, Kendall CB, Liang J: Automating carotid intima-media thickness video interpretation with convolutional neural networks.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016Google Scholar
  29. 29.
    Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition.. In: ICLR, 2015Google Scholar
  30. 30.
    Stark F, Hazırbas C, Triebel R., Cremers D.: Captcha recognition with active deep learning.. In: Workshop new challenges in neural computation 2015, p 94. Citeseer, 2015Google Scholar
  31. 31.
    Stein JH, Korcarz CE, Hurst RT, Lonn E, Kendall CB, Mohler ER, Najjar SS, Rembold CM, Post WS: Use of carotid ultrasound to identify subclinical vascular disease and evaluate cardiovascular disease risk: a consensus statement from the american society of echocardiography carotid intima-media thickness task force endorsed by the society for vascular medicine. J Am Soc Echocardiogr 21 (2): 93–111, 2008CrossRefGoogle Scholar
  32. 32.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A: Going deeper with convolutions.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2015Google Scholar
  33. 33.
    Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J: Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 35 (5): 1299–1312, 2016CrossRefGoogle Scholar
  34. 34.
    Tajbakhsh N, Shin JY, Hurst RT, Kendall CB, Liang J: Automatic interpretation of carotid intima–media thickness videos using convolutional neural networks.. In: Deep learning for medical image analysis, pp 105–131. Elsevier, 2017Google Scholar
  35. 35.
    Wang D, Shang Y: A new active labeling method for deep learning.. In: 2014 International joint conference on neural networks (IJCNN), pp 112–119, 2014.
  36. 36.
    Yang L, Zhang Y, Chen J, Zhang S, Chen DZ: Suggestive annotation: a deep active learning framework for biomedical image segmentation.. In: International conference on medical image computing and computer-assisted intervention, pp 399–407. Springer, 2017Google Scholar
  37. 37.
    Zhou Z, Shin J, Zhang L, Gurudu S, Gotway M, Liang J: Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally.. In: IEEE Conference on computer vision and pattern recognition, hawaii, pp 7340–7349, 2017Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2018

Authors and Affiliations

  1. 1.Arizona State UniversityScottsdaleUSA
  2. 2.Mayo ClinicScottsdaleUSA

Personalised recommendations