Computer-Aided Hepatocarcinoma Diagnosis Using Multimodal Deep Learning

  • Alan Baronio MenegottoEmail author
  • Carla Diniz Lopes Becker
  • Silvio Cesar Cazella
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1006)


Liver cancer was the fourth most deadly cancer in 2018 worldwide. Among liver cancers, hepatocarcinoma is the most prevalent cancer type. Diagnostic protocols are complex and suggest variations based on the patient’s context and the use of multiple data modalities. This paper briefly describes the steps involved in the development of a hepatocarcinoma computer-aided diagnosis using a multimodal deep learning approach with imaging and tabular data fusion. Data acquisition, preprocessing steps, architectural design decisions and possible use cases for the described architecture are discussed based on the partial results achieved on this ongoing research.


e-health Hepatocarcinoma Computer-aided diagnosis Multimodal deep learning 


  1. 1.
  2. 2.
    Stewart, B., Wild, C.: World Healt Report. International Agency for Research on Cancer, Lyon (2014)Google Scholar
  3. 3.
    Rahib, L., Smith, B.D., Aizenberg, R., Rosenzweig, A.B., Fleshman, J.M., Matrisian, L.M.: Projecting cancer incidence and deaths to 2030: the unexpected burden of thyroid, liver, and pancreas cancers in the united states. Cancer Res. 74, 2913–2921 (2014)CrossRefGoogle Scholar
  4. 4.
    Cancer Research UK: Worldwide cancer incidence statistics—Cancer Research UK (2014).
  5. 5.
    Attwa, M.H., El-Etreby, S.A.: Guide for diagnosis and treatment of hepatocellular carcinoma. World J. Hepatol. 7, 1632–1651 (2015)CrossRefGoogle Scholar
  6. 6.
    Bruix, J., Sherman, M.: Management of hepatocellular carcinoma: an update. Hepatol. 53, 1020–1022 (2011)CrossRefGoogle Scholar
  7. 7.
    Galle, P.R., Forner, A., Llovet, J.M., Mazzaferro, V., Piscaglia, F., Raoul, J.L., et al.: EASL clinical practice guidelines: management of hepatocellular carcinoma. J. Hepatol. 69, 182–236 (2018)CrossRefGoogle Scholar
  8. 8.
    Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017)CrossRefGoogle Scholar
  9. 9.
    Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRefGoogle Scholar
  10. 10.
    Esses, S.J., Lu, X., Zhao, T., Shanbhogue, K., Dane, B., Bruno, M., et al.: Automated image quality evaluation of T2-weighted liver MRI utilizing deep learning architecture. J. Magn. Reson. Imaging 47, 723–728 (2018)CrossRefGoogle Scholar
  11. 11.
    Ben-Cohen, A., Klang, E., Diamant, I., Rozendorn, N., Raskin, S.P., Konen, E., et al.: CT Image-based decision support system for categorization of liver metastases into primary cancer sites: initial results. Acad. Radiol. 24, 1501–1509 (2017)CrossRefGoogle Scholar
  12. 12.
    Sathurthi, S., Saruladha, K.: Prediction of liver cancer using random forest ensemble. Int. J. Pure Appl. Math. 116(21), 267–273 (2017)Google Scholar
  13. 13.
    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 689–696 (2011)Google Scholar
  14. 14.
    Ali, L., Khelil, K., Wajid, S.K., Hussain, Z.U., Shah, M.A., Howard, A., et al.: Machine learning based computer-aided diagnosis of liver tumours. In: 2017 IEEE 16th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC), pp. 139–145. IEEE (2017).
  15. 15.
    Wang, J., Jain, S., Chen, D., Song, W., Hu, C.T., Su, Y.H.: Development and evaluation of novel statistical methods in urine biomarker-based hepatocellular carcinoma screening. Sci. Rep. 8(1), 3799 (2018). Scholar
  16. 16.
    Dou, T., Zhang, L., Zheng, H., Zhou, W.: Local and non-local deep feature fusion for malignancy characterization of hepatocellular carcinoma, pp. 472–479. Springer, Cham (2018). Scholar
  17. 17.
    Wang, Q., Zhang, L., Xie, Y., Zheng, H., Zhou, W.: Malignancy characterization of hepatocellular carcinoma using hybrid texture and deep features. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 4162–4166. IEEE (2017).
  18. 18.
    Dou, T., Zhou, W.: 2D and 3D convolutional neural network fusion for predicting the histological grade of hepatocellular carcinoma. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3832–3837. IEEE (2018).
  19. 19.
    Erickson, B.J., Kirk, S., Lee, Y., Bathe, O., Kearns, M., Gerdes, C., et al.: TCGA-LIHC - The Cancer Imaging Archive (TCIA) Public Access - Cancer Imaging Archive Wiki (2016).
  20. 20.
    Lucchesi, F.R., Aredes, N.D.: Radiology data from the cancer genome atlas stomach adenocarcinoma [TCGA-STAD] collection (2016).
  21. 21.
    Linehan, M., Gautam, R., Kirk, S., Lee, Y., Roche, C., Bonaccio, E., et al.: Radiology data from the cancer genome atlas cervical kidney renal papillary cell carcinoma [KIRP] collection (2016).
  22. 22.
    National Cancer Institute Clinical Proteomic Tumor Analysis Consortium: Radiology data from the clinical proteomic tumor analysis consortium pancreatic ductal adenocarcinoma [CPTAC-PDA] collection (2018).
  23. 23.
    Pizer, S.M., Philip Amburn, E., Austin, J.D., Cromartie, R., Geselowitz, A., Greer, T., et al.: Adaptive histogram equalization and its variations. Comput. Vis. Graph. Image Process. 39, 355–368 (1987)CrossRefGoogle Scholar
  24. 24.
    Reza, A.M.: Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement. J. VLSI Sig. Process. Syst. Sig. Image Video Technol. 38, 35–44 (2004)CrossRefGoogle Scholar
  25. 25.
    Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20, 89–97 (2004)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103, 1449–1477 (2015)CrossRefGoogle Scholar
  27. 27.
    Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Sig. Process. Mag. 34, 96–108 (2017)CrossRefGoogle Scholar
  28. 28.
    Tajbakhsh, N., Shin, J.Y., Gurudu, S.R., Hurst, R.T., Kendall, C.B., Gotway, M.B., et al.: Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35, 1299–1312 (2016)CrossRefGoogle Scholar
  29. 29.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J.: Rethinking the inception architecture for computer vision (2016)Google Scholar
  30. 30.
  31. 31.
    Chollet, F.: Keras (2015).
  32. 32.
    Smith, R.A., Andrews, K.S., Brooks, D., Fedewa, S.A., Manassaram-Baptiste, D., Saslow, D., et al.: Cancer screening in the United States, 2018: a review of current American cancer society guidelines and current issues in cancer screening. CA Cancer J. Clin. 68, 297–316 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Alan Baronio Menegotto
    • 1
    Email author
  • Carla Diniz Lopes Becker
    • 1
  • Silvio Cesar Cazella
    • 1
  1. 1.Universidade Federal de Ciencias da Saude de Porto AlegrePorto AlegreBrazil

Personalised recommendations