Abstract
Although the application of deep learning has greatly improved the performance of benign and malignant breast cancer classification algorithm, the accuracy of classification using only the pathological image has been unable to meet the requirements of clinical practice. Inspired by the real scene when the pathologist read the pathological image for diagnosis, in this paper, we propose a new hybrid deep learning method for benign and malignant breast cancer classification. From the perspective of multimodal data fusion, our method combines pathological image and structured data in the clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification. Thus, the proposed method can be useful for breast cancer diagnosis in real clinical practice. Experimental results based on our datasets show that the proposed method significantly outperforms the state-of-the-art methods in terms of overall classification accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Holzinger, A., et al.: Towards the augmented pathologist: challenges of explainable-ai in digital pathology (2017)
Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)
Litjens, G., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42(9), 60–88 (2017)
Bayramoglu, N., Kannala, J., Heikkilä, J.: Deep learning for magnification independent breast cancer histopathology image classification. In: International Conference on Pattern Recognition (2017)
Xie, S., Tu, Z.: Holistically-Nested Edge Detection. Int. J. Comput. Vis. 125(1–3), 3–18 (2017)
Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34(6), 96–108 (2017)
Zhang, Z., Chen, P., Sapkota, M., Yang, L.: TandemNet: distilling knowledge from medical images using diagnostic reports as optional semantic references. In: Descoteaux, M., et al. (eds.) Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 MICCAI 2017. LNCS, vol. 10435, pp. 320–328. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_37
Xu, T., Zhang, H., Huang, X., Zhang, S., Metaxas, D.N.: Multimodal deep learning for cervical dysplasia diagnosis. In: Ourselin, S., Joskowicz, L., Sabuncu, M., Unal, G., Wells, W. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 MICCAI 2016. LNCS, vol. 9901, pp. 115–123. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_14
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous distributed systems (2016)
Spanhol, F.A., et al.: Breast cancer histopathological image classification using convolutional neural networks. In: International Joint Conference on Neural Networks (2016)
Araújo, T., et al.: Classification of breast cancer histology images using convolutional neural networks. PLoS ONE 12(6), e0177544 (2017)
Rakhlin, A., Shvets, A., Iglovikov, V., Kalinin, Alexandr A.: Deep convolutional neural networks for breast cancer histology image analysis. In: Campilho, A., Karray, F., ter Haar Romeny, B. (eds.) ICIAR 2018. LNCS, vol. 10882, pp. 737–744. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93000-8_83
Vang, Y.S., Chen, Z., Xie, X.: Deep learning framework for multi-class breast cancer histology image classification. In: Campilho, A., Karray, F., ter Haar Romeny, B. (eds.) ICIAR 2018. LNCS, vol. 10882, pp. 914–922. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93000-8_104
Golatkar, A., Anand, D., Sethi, A.: Classification of breast cancer histology using deep learning. In: Campilho, A., Karray, F., ter Haar Romeny, B. (eds.) ICIAR 2018. LNCS, vol. 10882, pp. 837–844. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93000-8_95
Aresta, G., et al.: BACH: grand challenge on breast cancer histology images (2018)
Acknowledgement
This research is supported by the National Key Research and Development Program of China (No. 2017YFE0103900 and 2017YFA0504702), the NSFC projects Grant (No. U1611263, U1611261, 61502455 and 61672493), Peking University International Hospital Research Grant (No. YN2018ZD05), Beijing Municipal Natural Science Foundation Grant (No. L182053) and Special Program for Applied Research on Super Computation of the NSFC-Guangdong Joint Fund (the second phase).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Yan, R. et al. (2019). Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method. In: Huang, DS., Bevilacqua, V., Premaratne, P. (eds) Intelligent Computing Theories and Application. ICIC 2019. Lecture Notes in Computer Science(), vol 11643. Springer, Cham. https://doi.org/10.1007/978-3-030-26763-6_44
Download citation
DOI: https://doi.org/10.1007/978-3-030-26763-6_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-26762-9
Online ISBN: 978-3-030-26763-6
eBook Packages: Computer ScienceComputer Science (R0)