Combining Convolutional and Recurrent Neural Networks for Classification of Focal Liver Lesions in Multi-phase CT Images

  • Dong Liang
  • Lanfen LinEmail author
  • Hongjie Hu
  • Qiaowei Zhang
  • Qingqing Chen
  • Yutaro lwamoto
  • Xianhua Han
  • Yen-Wei Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11071)


Computer-aided diagnosis (CAD) systems are useful for assisting radiologists with clinical diagnoses by classifying focal liver lesions (FLLs) based on multi-phase computed tomography (CT) images. Although many studies have conducted in the field, there still remain two challenges. First, the temporal enhancement pattern is hard to represent effectively. Second, the local and global information of lesions both are necessary for this task. In this paper, we proposed a framework based on deep learning, called ResGL-BDLSTM, which combines a residual deep neural network (ResNet) with global and local pathways (ResGL Net) with a bi-directional long short-term memory (BD-LSTM) model for the task of focal liver lesions classification in multi-phase CT images. In addition, we proposed a novel loss function to train the proposed framework. The loss function is composed of an inter-loss and intra-loss, which can improve the robustness of the framework. The proposed framework outperforms state-of-the-art approaches by achieving a 90.93% mean accuracy.


Deep learning ResGLNet BD-LSTM Liver lesions classification Computer-aid diagnosis (CAD) system 



This work was supported in part by the National Key R&D Program of China under the Grant No. 2017YFB0309800, in part by the Key Science and Technology Innovation Support Program of Hangzhou under the Grant No. 20172011A038, and in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant No. 18H03267 and No. 17H00754.


  1. 1.
    Ryerson, A.B., et al.: Annual report to the nation on the status of cancer, 1975–2012, featuring the increasing incidence of liver cancer. Cancer 122(9), 1312–1337 (2016)CrossRefGoogle Scholar
  2. 2.
    Chen, J., et al.: Combining fully convolutional and recurrent neural networks for 3D biomedical image segmentation. In: Advances in Neural Information Processing Systems (2016)Google Scholar
  3. 3.
    Roy, S., et al.: Three-dimensional spatiotemporal features for fast content-based retrieval of focal liver lesions. IEEE Trans. Biomed. Eng. 61(11), 2768–2778 (2014)CrossRefGoogle Scholar
  4. 4.
    Yu, M., et al.: Extraction of lesion-partitioned features and retrieval of contrast-enhanced liver images. Comput. Math. Meth. Med. 2012, 12 (2012)Google Scholar
  5. 5.
    Yang, W., et al.: Content-based retrieval of focal liver lesions using bag-of-visual-words representations of single-and multiphase contrast-enhanced CT images. J. Digital Imaging 25(6), 708–719 (2012)CrossRefGoogle Scholar
  6. 6.
    Diamant, I., et al.: Improved patch-based automated liver lesion classification by separate analysis of the interior and boundary regions. IEEE J. Biomed. Health Inform. 20(6), 1585–1594 (2016)CrossRefGoogle Scholar
  7. 7.
    Xu, Y., et al.: Bag of temporal co-occurrence words for retrieval of focal liver lesions using 3D multiphase contrast-enhanced CT images. In: Proceedings of 23rd International Conference on Pattern Recognition (ICPR 2016), pp. 2283–2288 (2016)Google Scholar
  8. 8.
    Wang J., et al.: Sparse codebook model of local structures for retrieval of focal liver lesions using multiphase medical images. Int. J. Biomed. Imaging 2017, 13 p. (2017)Google Scholar
  9. 9.
    Xu, Y., et al.: Texture-specific bag of visual words model and spatial cone matching-based method for the retrieval of focal liver lesions using multiphase contrast-enhanced CT images. Int. J. Comput. Assist. Radiol. Surg. 13(1), 151–164 (2018)CrossRefGoogle Scholar
  10. 10.
    He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  11. 11.
    Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., Greenspan, H.: Modeling the intra-class variability for liver lesion detection using a multi-class patch-based CNN. In: Wu, G., Munsell, Brent C., Zhan, Y., Bai, W., Sanroma, G., Coupé, P. (eds.) Patch-MI 2017. LNCS, vol. 10530, pp. 129–137. Springer, Cham (2017). Scholar
  12. 12.
    Yasaka, K., et al.: Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology 286(3), 887–896 (2017)CrossRefGoogle Scholar
  13. 13.
    Liang, D., et al.: Residual convolutional neural networks with global and local pathways for classification of focal liver lesions. In: Geng, X., Kang, B.H. (eds.) PRICAI 2018: Trends in Artificial Intelligence. PRICAI 2018. LNCS, vol. 11012, pp. 617–628. Springer, Cham (2018). Scholar
  14. 14.
    Dong, C., et al.: Simultaneous segmentation of multiple organs using random walks. J. Inf. Process. 24(2), 320–329 (2016)Google Scholar
  15. 15.
    Dong, C., et al.: Non-rigid image registration with anatomical structure constraint for assessing locoregional therapy of hepatocellular carcinoma. Comput. Med. Imaging Graph. 45, 75–83 (2015)CrossRefGoogle Scholar
  16. 16.
    Wen, Y., Zhang, K., Li, Z., Qiao, Yu.: A discriminative feature learning approach for deep face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 499–515. Springer, Cham (2016). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Dong Liang
    • 1
  • Lanfen Lin
    • 1
    Email author
  • Hongjie Hu
    • 2
  • Qiaowei Zhang
    • 2
  • Qingqing Chen
    • 2
  • Yutaro lwamoto
    • 3
  • Xianhua Han
    • 3
  • Yen-Wei Chen
    • 3
    • 1
  1. 1.Department of Computer Science and TechnologyZhejiang UniversityHangzhouChina
  2. 2.Department of RadiologySir Run Run Shaw HospitalHangzhouChina
  3. 3.College of Information Science and EngineeringRitsumeikan UniversityKusatsuJapan

Personalised recommendations