Skip to main content
SpringerLink
Log in

Multimodal Deep Learning for Computer-Aided Detection and Diagnosis of Cancer: Theory and Applications

  • Chapter
  • First Online:
Book cover Enhanced Telemedicine and e-Health

Part of the book series: Studies in Fuzziness and Soft Computing ((STUDFUZZ,volume 410))

Abstract

Cancer is a group of diseases caused by the abnormal and disorderly growth of cells, representing the second leading cause of deaths worldwide. The number of cancer cases is growing yearly, medical systems are an essential tool to speed up the diagnosis process and increase patient survival probabilities. Electronic health record systems store the patient’s health data, which can be of structured and unstructured types. Physicians use all the information available in these systems during a cancer diagnostic, regardless of its sort and modality. Deep learning is a machine learning sub-field that has algorithms able to process data end-to-end using deep architectures, often inspired by the brain’s synaptic model. Using more than one data source in these architectures is known as multimodal deep learning. Computer-aided detection and diagnosis systems developed using multimodal deep learning algorithms have been achieving promising results in diagnostic performance, which are often comparable to human specialists. E-health and telemedicine applications can be boosted with these high-performance detection and diagnosis systems, which can provide real-time analysis capabilities to the healthcare staff and improve the quality of the medical services. This chapter explores the theory and the applications of multimodal deep learning techniques to develop computer-aided detection and diagnosis systems for cancer, describing examples of the state-of-the-art in this area, looking for essential and innovative aspects of these systems. Heterogeneous and hybrid fusion strategies combining high-quality imaging, clinical attributes and genetic data in deep multimodal architectures are resulting in computer-aided systems for cancer with promising diagnosis performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. F. Bray, J. Ferlay, I. Soerjomataram, R.L. Siegel, L.A. Torre, A. Jemal, Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. Cancer J. Clin. 68, 394–424 (2018)

    Article  Google Scholar 

  2. International Agency for Research on Cancer. Cancer Tomorrow. Available at: https://gco.iarc.fr/tomorrow/home. Accessed 27 Aug 2020

  3. Cancer Research UK. Worldwide cancer incidence statistics. Available at: https://www.cancerresearchuk.org/health-professional/cancer-statistics/worldwide-cancer/incidence%5C#heading-Five. Accessed 27 Aug 2020

  4. P.C. Valery, M. Laversanne, P.J. Clark, J.L. Petrick, K.A. McGlynn, F. Bray, Projections of primary liver cancer to 2030 in 30 countries worldwide. Hepatology 67(2), 600–611 (2018)

    Article  Google Scholar 

  5. E.C. Ellison, T.M. Pawlik, D.P. Way, B. Satiani, T.E. Williams, The impact of the aging population and incidence of cancer on future projections of general surgical workforce needs. Surgery 163(3), 553–559 (2018)

    Article  Google Scholar 

  6. S. Price, B. Golden, E. Wasil, B.T. Denton, Operations research models and methods in the screening, detection, and treatment of prostate cancer: a categorized, annotated review. Oper. Res. Health Care 8, 9–21 (2016)

    Article  Google Scholar 

  7. A. Esteva, B. Kuprel, R.A. Novoa, J. Ko, S.M. Swetter, H.M. Blau, S. Thrun, Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)

    Article  Google Scholar 

  8. Y. Xu, Deep learning in multimodal medical image analysis, in International Conference on Health Information Science (2019), pp. 193–200

    Google Scholar 

  9. G.S. Lodwick, C.L. Haun, W.E. Smith, R.F. Keller, E.D. Robertson, Computer diagnosis of primary bone tumors: a preliminary report. Radiology 80(2), 273–275 (1963)

    Article  Google Scholar 

  10. F.T. De Dombal, D.J. Leaper, J.R. Staniland, A.P. McCann, J.C. Horrocks, Computer-aided diagnosis of acute abdominal pain. Br. Med. J. 2(5804), 9–13 (1972)

    Article  Google Scholar 

  11. K. Doi, Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 31(4–5), 198–211 (2007)

    Article  Google Scholar 

  12. J. Roehrig, T. Doi, A. Hasegawa, B. Hunt, J. Marshall, H. Romsdahl, A. Schneider, R. Sharbaugh, W. Zhang, Clinical results with R2 ImageChecker system. Digital Mammography, 395–400 (1998)

    Google Scholar 

  13. L. Vassallo, A. Traverso, M. Agnello, C. Bracco, D. Campanella, G. Chiara, M.E. Fantacci, E.L. Torres, A. Manca, M. Saletta, V. Giannini, A cloud-based computer-aided detection system improves identification of lung nodules on computed tomography scans of patients with extra-thoracic malignancies. Eur. Radiol. 29(1), 144–152 (2019)

    Article  Google Scholar 

  14. H. Fujita, D. Cimr, Computer aided detection for fibrillations and flutters using deep convolutional neural network. Inf. Sci. 486, 231–239 (2019)

    Article  Google Scholar 

  15. H. Abdeltawab, M. Shehata, A. Shalaby, F. Khalifa, A. Mahmoud, M. Abou El-Ghar, A.C. Dwyer, M. Ghazal, H. Hajjdiab, R. Keynton, A. El-Baz, A novel CNN-based CAD system for early assessment of transplanted kidney dysfunction. Sci. Rep. 9(1), 1–11 (2019)

    Article  Google Scholar 

  16. F. Ayatollahi, S.B. Shokouhi, J. Teuwen, Differentiating benign and malignant mass and non-mass lesions in breast DCE-MRI using normalized frequency-based features. Int. J. Comput. Assist. Radiol. Surg. 15(2), 297–307 (2020)

    Article  Google Scholar 

  17. Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  18. A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (2012), pp. 1097–1105

    Google Scholar 

  19. G. Litjens, T. Kooi, B.E. Bejnordi, A.A.A. Setio, F. Ciompi, M. Ghafoorian, J.A. Van Der Laak, B. Van Ginneken, C.I. Sánchez, A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)

    Article  Google Scholar 

  20. D. Sun, M. Wang, A. Li, A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data. IEEE/ACM Trans. Comput. Biol. Bioinf. 16(3), 841–850 (2018)

    Article  Google Scholar 

  21. T. Dou, L. Zhang, H. Zheng, W. Zhou, Local and non-local deep feature fusion for malignancy characterization of hepatocellular carcinoma, in International Conference on Medical Image Computing and Computer-Assisted Intervention (2018), pp. 472–479

    Google Scholar 

  22. P.R. Galle, A. Forner, J.M. Llovet, V. Mazzaferro, F. Piscaglia, J.L. Raoul, P. Schirmacher, V. Vilgrain, EASL clinical practice guidelines: management of hepatocellular carcinoma. J. Hepatol. 69(1), 182–236 (2018)

    Article  Google Scholar 

  23. F.E. White, Data fusion lexicon. Tech. Rep. Joint Directors of Labs Washington DC (1991)

    Google Scholar 

  24. D. Lahat, T. Adali, C. Jutten, Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015)

    Article  Google Scholar 

  25. F. Castanedo, A review of data fusion techniques. Sci. World J. 704504 (2013)

    Google Scholar 

  26. M. Oral, S.S. Turgut, A comparative study for image fusion, in IEEE Innovations in Intelligent Systems and Applications Conference (2018), pp. 1–6

    Google Scholar 

  27. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A.Y. Ng, Multimodal deep learning, in Proceedings of the 28th International Conference on International Conference on Machine Learning (2011), pp. 689–696

    Google Scholar 

  28. D. Ramachandram, G.W. Taylor, Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34(6), 96–108 (2017)

    Article  Google Scholar 

  29. A. Shrestha, A. Mahmood, Review of deep learning algorithms and architectures. IEEE Access 7, 53040–53065 (2019)

    Article  Google Scholar 

  30. S.J. Russell, P. Norvig, E. Davis, Artificial Intelligence: A Modern Approach, 3rd edn. (2010), pp. 693–748

    Google Scholar 

  31. I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, vol. 1 (MIT press, Cambridge, 2016), pp. 326–366

    Google Scholar 

  32. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 2818–2826

    Google Scholar 

  33. S. Xie, R. Girshick, P. Dollár, Z. Tu, K. He, Aggregated residual transformations for deep neural networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 1492–1500

    Google Scholar 

  34. J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 7132–7141

    Google Scholar 

  35. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, Greedy layer-wise training of deep networks, in Advances in Neural Information Processing Systems (2007), pp. 153–160

    Google Scholar 

  36. I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, vol. 1 (MIT press, Cambridge, 2016), pp. 502–524

    Google Scholar 

  37. Z. Guo, X. Li, H. Huang, N. Guo, Q. Li, Deep learning-based image segmentation on multimodal medical imaging. IEEE Trans. Radiat. Plasma Med. Sci. 3(2), 162–169 (2019)

    Article  Google Scholar 

  38. K. Munir, H. Elahi, A. Ayub, F. Frezza, A. Rizzi, Cancer diagnosis using deep learning: a bibliographic review. Cancers 11(9), 1235 (2019)

    Article  Google Scholar 

  39. A.B. Menegotto, C.D.L. Becker, S.C. Cazella, Computer-aided hepatocarcinoma diagnosis using multimodal deep learning. Ambient Intell. Softw. Appl. 1006, 3–10 (2019)

    Google Scholar 

  40. Harvard Medical School. Brain Tumor Overview. Available at: https://www.health.harvard.edu/a%5C_to%5C_z/brain-tumor-overview-a-to-z. Accessed 27 Aug 2020

  41. Y. Li, L. Shen, Deep learning based multimodal brain tumor diagnosis. Int. MICCAI Brain Lesion Workshop (2017), pp. 149–158

    Google Scholar 

  42. J. Amin, M. Sharif, N. Gul, M. Raza, M.A. Anjum, M.W. Nisar, S.A.C. Bukhari, Brain tumor detection by using stacked autoencoders in deep learning. J. Med. Syst. 44(2), 32 (2020)

    Article  Google Scholar 

  43. T. Saba, A.S. Mohamed, M. El-Affendi, J. Amin, M. Sharif, Brain tumor detection using fusion of hand crafted and deep learning features. Cogn. Syst. Res. 59, 221–230 (2020)

    Article  Google Scholar 

  44. American Cancer Society. Lung Cancer Overview. Available at: https://www.cancer.org/cancer/lung-cancer/about.html. Accessed 27 Aug 2020

  45. H. Shi, N. Zhang, X.Q. Wu, Y.D. Zhang, Multimodal lung tumor image recognition algorithm based on integrated convolutional neural network. Concurrency Comput. Pract. Experience (2018), pp. e4965

    Google Scholar 

  46. L. Yu-Heng, C. Wei-Ning, H. Te-Cheng, C. Lin, Y. Tsao, W. Semon, Overall survival prediction of non-small cell lung cancer by integrating microarray and clinical data with deep learning. Sci. Rep. 10(1) (2020)

    Google Scholar 

  47. R. Qin, Z. Wang, L. Jiang, K. Qiao, J. Hai, J. Chen, J. Xu, D. Shi, B. Yan, Fine-grained lung cancer classification from PET and CT images based on multidimensional attention mechanism. Complexity 2020 (2020)

    Google Scholar 

  48. G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected convolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 4700–4708

    Google Scholar 

  49. J. Arevalo, T. Solorio, M. Montes-y-Gomez, F.A. González, Gated multimodal networks. Neural Comput. Appl. 1–20 (2020)

    Google Scholar 

  50. J. Balogh, D. Victor III, E.H. Asham, S.G. Burroughs, M. Boktour, A. Saharia, X. Li, R.M. Ghobrial, H.P. Monsour Jr., Hepatocellular carcinoma: a review. J. Hepatocellular Carcinoma 3, 41 (2016)

    Article  Google Scholar 

  51. W.C. Tsai, P.T. Kung, Y.H. Wang, W.Y. Kuo, Y.H. Li, Influence of the time interval from diagnosis to treatment on survival for early-stage liver cancer. PLoS ONE 13(6), e0199532 (2018)

    Article  Google Scholar 

  52. J.A. Marrero, L.M. Kulik, C.B. Sirlin, A.X. Zhu, R.S. Finn, M.M. Abecassis, L.R. Roberts, J.K. Heimbach, Diagnosis, staging, and management of hepatocellular carcinoma: 2018 practice guidance by the American Association for the study of liver diseases. Hepatology 68(2), 723–750 (2018)

    Article  Google Scholar 

  53. R.F. Hanna, Z.M. Vesselin, T. An, A.F. Lee, Z.B. Sidney, S.S. Ranjit, S.S. Cynthia, W. Tanya, G. Anthony, B.S. Claude, Comparative 13-year meta-analysis of the sensitivity and positive predictive value of ultrasound, CT, and MRI for detecting hepatocellular carcinoma. Abdom. Radiol. 41(1), 71–90 (2016)

    Article  Google Scholar 

  54. H. Lin, C. Wei, G. Wang, H. Chen, L. Lin, M. Ni, J. Chen, S. Zhuo, Automated classification of hepatocellular carcinoma differentiation using multiphoton microscopy and deep learning. J. Biophotonics 12, e201800435 (2019)

    Google Scholar 

  55. K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in 3rd International Conference on Learning Representations (2015)

    Google Scholar 

  56. S.H. Zhen, M. Cheng, Y.B. Tao, Y.F. Wang, S. Juengpanich, Z.Y. Jiang, Y.K. Jiang, Y.Y. Yan, W. Lu, J.M. Lue, J.H. Qian, Deep learning for accurate diagnosis of liver tumor based on magnetic resonance imaging and clinical data. Front. Oncol. 10, 680 (2020)

    Article  Google Scholar 

  57. I. Reda, A. Khalil, M. Elmogy, A. Abou El-Fetouh, A. Shalaby, M. Abou El-Ghar, A. Elmaghraby, M. Ghazal, A. El-Baz, Deep learning role in early diagnosis of prostate cancer. Technol. Cancer Res. Treat. 17, 153 (2018)

    Article  Google Scholar 

  58. B. Song, S. Sunny, R.D. Uthoff, S. Patrick, A. Suresh, T. Kolur, G. Keerthi, A. Anbarani, Automatic classification of dual-modalilty, smartphone-based oral dysplasia and malignancy images using deep learning. Biomed. Optics Expr. 9(11), 5318–5329 (2018)

    Article  Google Scholar 

  59. M.R. Karim, G. Wicaksono, I.G. Costa, S. Decker, O. Beyan, Prognostically relevant subtypes and survival prediction for breast cancer based on multimodal genomics data. IEEE Access 7, 133850–133864 (2019)

    Article  Google Scholar 

  60. N. Jaques, S. Taylor, A. Sano, R. Picard, Multimodal autoencoder: a deep learning approach to filling in missing sensor data and enabling better mood prediction, in Seventh International Conference on Affective Computing and Intelligent Interaction (2017), pp. 202–208

    Google Scholar 

  61. S. Ding, H. Huang, Z. Li, X. Liu, S. Yang, SCNET: a novel UGI cancer screening framework based on semantic-level multimodal data fusion. IEEE J. Biomed. Health Inf. (2020)

    Google Scholar 

  62. T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems (2013), pp. 3111–3119

    Google Scholar 

  63. A. Cheerla, O. Gevaert, Deep learning with multimodal representation for pan-cancer prognosis prediction. Bioinformatics 35, i446–i454 (2019)

    Article  Google Scholar 

  64. R.K. Srivastava, K. Greff, J. Schmidhuber, Training very deep networks, in Advances in Neural Information Processing Systems (2015), pp. 2377–2385

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. UFCSPA, Porto Alegre, Brazil

    Alan Baronio Menegotto & Silvio Cesar Cazella

Authors
  1. Alan Baronio Menegotto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Silvio Cesar Cazella
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alan Baronio Menegotto .

Editor information

Editors and Affiliations

  1. Polytechnic of Coimbra, ESTGOH, Rua General Santos Costa, Oliveira do Hospital, Coimbra, Portugal

    Gonçalo Marques

  2. Department of Electrical and Electronics Engineering, Sikkim Manipal Institute of Technology, Majhitar, Sikkim, India

    Akash Kumar Bhoi

  3. University of Valladolid, Valladolid, Spain

    Isabel de la Torre Díez

  4. Universidad de Deusto, Bilbao, Vizcaya, Spain

    Begonya Garcia-Zapirain

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Menegotto, A.B., Cazella, S.C. (2021). Multimodal Deep Learning for Computer-Aided Detection and Diagnosis of Cancer: Theory and Applications. In: Marques, G., Kumar Bhoi, A., de la Torre Díez, I., Garcia-Zapirain, B. (eds) Enhanced Telemedicine and e-Health. Studies in Fuzziness and Soft Computing, vol 410. Springer, Cham. https://doi.org/10.1007/978-3-030-70111-6_13

Download citation

Publish with us

Policies and ethics

Search

Navigation