Abstract
Breast cancer is the most common invasive cancer in women. Besides the primary B-mode ultrasound screening, sonographers have explored the inclusion of Doppler, strain and shear-wave elasticity imaging to advance the diagnosis. However, recognizing useful patterns in all types of images and weighing up the significance of each modality can elude less-experienced clinicians. In this paper, we explore, for the first time, an automatic way to combine the four types of ultrasonography to discriminate between benign and malignant breast nodules. A novel multimodal network is proposed, along with promising learnability and simplicity to improve classification accuracy. The key is using a weight-sharing strategy to encourage interactions between modalities and adopting an additional cross-modalities objective to integrate global information. In contrast to hardcoding the weights of each modality in the model, we embed it in a Reinforcement Learning framework to learn this weighting in an end-to-end manner. Thus the model is trained to seek the optimal multimodal combination without handcrafted heuristics. The proposed framework is evaluated on a dataset contains 1616 sets of multimodal images. Results showed that the model scored a high classification accuracy of 95.4%, which indicates the efficiency of the proposed method.
J. Wang and J. Miao—Contribute equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ciatto, S., Cecchini, S., Lossa, A., Grazzini, G.: Category and operable breast cancer prognosis. Tumori J. 75(1), 18–22 (1989)
World Health Organization (WHO). breast cancer. www.who.int/cancer/prevention/diagnosis-screening/breastcancer/. Accessed 2019
Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2018)
Ge, Z., Demyanov, S., Chakravorty, R., Bowling, A., Garnavi, R.: Skin disease recognition using deep saliency features and multimodal learning of dermoscopy and clinical images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 250–258. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_29
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Jesneck, J.L., Lo, J.Y., Baker, J.A.: Breast mass lesions: computer-aided diagnosis models with mammographic and sonographic descriptors. Radiology 244(2), 390–398 (2007)
Li, C., et al.: AM-LFS: AutoML for loss function search. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8410–8419 (2019)
Liu, J., et al.: Integrate domain knowledge in training CNN for ultrasonography breast cancer diagnosis. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 868–875. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_96
Morvant, E., Habrard, A., Ayache, S.: Majority vote of diverse classifiers for late fusion. In: Fränti, P., Brown, G., Loog, M., Escolano, F., Pelillo, M. (eds.) S+SSPR 2014. LNCS, vol. 8621, pp. 153–162. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44415-3_16
Murtaza, G., et al.: Deep learning-based breast cancer classification through medical imaging modalities: state of the art and research challenges. Artif. Intell. Rev. 53, 1–66 (2019)
Sultan, L.R., Cary, T.W., Sehgal, C.M.: Machine learning to improve breast cancer diagnosis by multimodal ultrasound. In: 2018 IEEE International Ultrasonics Symposium (IUS), pp. 1–4. IEEE (2018)
Vielzeuf, V., Lechervy, A., Pateux, S., Jurie, F.: CentralNet: a multilayer approach for multimodal fusion. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11134, pp. 575–589. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_44
Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992). https://doi.org/10.1007/BF00992696
Zhang, Q., Song, S., Xiao, Y., Chen, S., Shi, J., Zheng, H.: Dual-mode artificially-intelligent diagnosis of breast tumours in shear-wave elastography and b-mode ultrasound using deep polynomial networks. Med. Eng. Phys. 64, 1–6 (2019)
Zhang, Q., et al.: Deep learning based classification of breast tumors with shear-wave elastography. Ultrasonics 72, 150–157 (2016)
Acknowledgement
This work was supported by National Key R&D Program of China (No. 2019YFC 0118300); Shenzhen Peacock Plan (No. KQTD2016053112051497, KQJSCX2018 0328095606003); Medical Scientific Research Foundation of Guangdong Province, China (No. B2018031); National Natural Science Foundation of China (Project No. NSFC61771130).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J. et al. (2020). Auto-weighting for Breast Cancer Classification in Multimodal Ultrasound. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12266. Springer, Cham. https://doi.org/10.1007/978-3-030-59725-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-59725-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59724-5
Online ISBN: 978-3-030-59725-2
eBook Packages: Computer ScienceComputer Science (R0)
