Abstract
Capturing global contextual information plays a critical role in breast ultrasound (BUS) image classification. Although convolutional neural networks (CNNs) have demonstrated reliable performance in tumor classification, they have inherent limitations for modeling global and long-range dependencies due to the localized nature of convolution operations. Vision Transformers have an improved capability of capturing global contextual information but may distort the local image patterns due to the tokenization operations. In this study, we proposed a hybrid multitask deep neural network called Hybrid-MT-ESTAN, designed to perform BUS tumor classification and segmentation using a hybrid architecture composed of CNNs and Swin Transformer components. The proposed approach was compared to nine BUS classification methods and evaluated using seven quantitative metrics on a dataset of 3,320 BUS images. The results indicate that Hybrid-MT-ESTAN achieved the highest accuracy, sensitivity, and F1 score of 82.7%, 86.4%, and 86.0%, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
American Cancer Society, Cancer Facts & Figures, https://www.cancer.org, (2022)
Zhuang, Z., Yang, Z., Raj, A., Noel, J., Wei, C.: Breast ultrasound tumor image classification using image decomposition and fusion based on adaptive multi-model spatial feature fusion. Comput. Methods Programs Biomed. 208, 106221 (2021)
Shareef, B., et al.: A benchmark for breast ultrasound image classification. Available at SSRN (2023). https://ssrn.com/abstract=4339660
Iqbal, A., Sharif, M., BTS-ST: Swin transformer network for segmentation and classification of multimodality breast cancer images. Knowl. Based Syst. 262, 110393 (2023)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale, preprint arXiv:2010.11929 (2020)
Yap, M., et al.: Automated breast ultrasound lesions detection using convolutional neural networks. IEEE J. Biomed. Health Inf. 22(4), 1218–1226 (2017)
Huang, G., Liu, Z., Mateen, L., Weinberger, K.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700–4708 (2017)
Zhang, G., Zhao, K., Hong, Y., Qiu, X., Zhang, K., Wei, B.: SHA-MTL: soft and hard attention multi-task learning for automated breast cancer ultrasound image segmentation and classification. Int. J. Comput. Assisted Radiol. Surg. 16(10), 1719–1725 (2021). https://doi.org/10.1007/s11548-021-02445-7
Geertsma, T., Fujifilm.: Ultrasound cases, https://www.ultrasoundcases.info/ (2014)
Chowdary, J., Yogarajah, P., Chaurasia, P., Guruviah, V.: A multi-task learning framework for automated segmentation and classification of breast tumors from ultrasound images. Ultrason. Imaging 44(1), 3–12 (2022)
Vakanski, A., Xian, M.: Evaluation of complexity measures for deep learning generalization in medical image analysis. In: 2021 IEEE 31st Int. Workshop on MLSP, pp. 1–6 (2021)
Shi, J., Vakanski, A., Xian, M., Ding, J., Ning, C.: EMT-NET: efficient multitask network for computer-aided diagnosis of breast cancer. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5 (2022)
Gheflati, B., Rivaz, H.: Vision transformers for classification of breast ultrasound images, In: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 480–483 (2022)
Hassanien, A., Singh, K., Puig, D., Abdel-Nasser, M.: Predicting breast tumor malignancy using deep ConvNeXt radiomics and quality-based score pooling in ultrasound sequences. Diagnostics 12(5), 1053 (2022)
Mo, Y., et al.: Hover-trans: anatomy-aware hover-transformer for ROI-Free breast cancer diagnosis in ultrasound images IEEE Trans. Med. Imaging 42, 1696–1706 (2023)
Qu, X., et al.: A VGG attention vision transformer network for benign and malignant classification of breast ultrasound images. Med. Phys. 49(9), 5787–5798 (2022)
Shareef, B., Vakanski, A., Freer, P., Min Xian: ESTAN: enhanced small tumor-aware network for breast ultrasound image segmentation. Healthcare 10(11), 2262 (2022)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T.: MobileNets: efficient convolutional neural networks for mobile vision applications. preprint arXiv:1704.04861 (2017)
Zhang, Y., et al.: BUSIS: a benchmark for breast ultrasound image segmentation. Healthcare 10(4), 729 (2022)
Yap, M., et al.: Breast ultrasound lesions recognition: end-to-end deep learning approaches. J. Med. Imaging 6(1), 011007. SPIE (2018)
Shareef, B., Xian, M., Vakanski, A.: Stan: small tumor-aware network for breast ultrasound image segmentation. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 1–5 (2020)
Ayana, G., Choe, S.: BUViTNet: breast ultrasound detection via vision transformers, Diagnostics 12(11), 2654 (2022). https://doi.org/10.3390/diagnostics12112654
Tang, S., et al.: Transformer-based multi-task learning for classification and segmentation of gastrointestinal tract endoscopic images. Comput. Bio. Med. 157, 106723 (2023)
Sebastian, R.: An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 (2017)
Acknowledgement
Research reported in this publication was supported by the National Institute Of General Medical Sciences of the National Institutes of Health under Award Number P20GM104420. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shareef, B., Xian, M., Vakanski, A., Wang, H. (2023). Breast Ultrasound Tumor Classification Using a Hybrid Multitask CNN-Transformer Network. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14223. Springer, Cham. https://doi.org/10.1007/978-3-031-43901-8_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-43901-8_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43900-1
Online ISBN: 978-3-031-43901-8
eBook Packages: Computer ScienceComputer Science (R0)
