Skip to main content

Advertisement

SpringerLink
Log in

Vision Transformers-Based Transfer Learning for Breast Mass Classification From Multiple Diagnostic Modalities

  • Original Article
  • Published:
Journal of Electrical Engineering & Technology Aims and scope Submit manuscript

Abstract

Breast mass evaluation is crucial for early breast cancer diagnosis via imaging. While Convolutional Neural Network (CNN)-based deep learning (DL) has enhanced this process, it suffers from computational complexity and limited spatial encoding. Vision Transformer (ViT)-based DL, more adept at encoding spatial information, presents a promising alternative. This study introduces a ViT-based transfer learning (TL) method for breast mass classification. Three ViT-based TL architectures pretrained on ImageNet were proposed and evaluated using ultrasound and mammogram datasets. Comparative analysis against ViT trained from scratch and CNN-based TL was conducted. Results showed the ViT-based TL method achieving the highest area under curve (AUC) of 1 ± 0 for both datasets, outperforming ViT from scratch and yielding similar or better performance compared to CNN-based TL. Despite its computational cost, ViT-based TL demonstrates superior classification capabilities for breast mass images. This research provides a foundational framework for future studies exploring ViT-based TL in breast cancer diagnosis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

In this study, we used publicly available mammogram datasets from Mendeley Data (https://data.mendeley.com/datasets/ywsbh3ndr8/2, accessed last on July 10, 2023) and ultrasound breast images from Mendeley Data (https://data.mendeley.com/datasets/wmy84gzngw/1, accessed last on July 10, 2023) and Kaggle (https://www.kaggle.com/datasets/aryashah2k/breast-ultrasoundimages-dataset, accessed last on July 10, 2023).

References

  1. Sung H, Ferlay J, Siegel RL et al (2021) Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin 71:209–249. https://doi.org/10.3322/caac.21660

    Article  Google Scholar 

  2. McCormack V, McKenzie F, Foerster M et al (2020) Breast cancer survival and survival gap apportionment in sub-Saharan Africa (ABC-DO): a prospective cohort study. Lancet Glob Heal 8:e1203–e1212. https://doi.org/10.1016/S2214-109X(20)30261-8

    Article  Google Scholar 

  3. Seely JM, Alhassan T (2018) Screening for Breast Cancer in 2018—What Should We be Doing Today? Curr Oncol 25:115–124. https://doi.org/10.3747/co.25.3770

    Article  Google Scholar 

  4. Liu H, Zhan H, Sun D, Zhang Y (2020) Comparison of BSGI, MRI, mammography, and ultrasound for the diagnosis of breast lesions and their correlations with specific molecular subtypes in Chinese women. BMC Med Imaging 20:1–10. https://doi.org/10.1186/s12880-020-00497-w

    Article  Google Scholar 

  5. Ayana G, Ryu J, Choe S (2022) Ultrasound-Responsive Nanocarriers for Breast Cancer Chemotherapy. Micromachines 13:1508. https://doi.org/10.3390/mi13091508

    Article  Google Scholar 

  6. Ayana G, Dese K, Raj H et al (2022) De-Speckling Breast Cancer Ultrasound Images Using a Rotationally Invariant Block Matching Based Non-Local Means (RIBM-NLM) Method. Diagnostics 12:862. https://doi.org/10.3390/diagnostics12040862

    Article  Google Scholar 

  7. Woods RW, Sisney GS, Salkowski LR et al (2011) The Mammographic Density of a Mass Is a Significant Predictor of Breast Cancer. Radiology 258:417–425. https://doi.org/10.1148/radiol.10100328

    Article  Google Scholar 

  8. Aboutalib SS, Mohamed AA, Berg WA et al (2018) Deep learning to distinguish recalled but benign mammography images in breast cancer screening. Clin Cancer Res 24:5902–5909. https://doi.org/10.1158/1078-0432.CCR-18-1115

    Article  Google Scholar 

  9. Giampietro RR, Cabral MVG, Lima SAM et al (2020) Accuracy and Effectiveness of Mammography versus Mammography and Tomosynthesis for Population-Based Breast Cancer Screening: A Systematic Review and Meta-Analysis. Sci Rep 10:7991. https://doi.org/10.1038/s41598-020-64802-x

    Article  Google Scholar 

  10. Zheng J, Lin D, Gao Z et al (2020) Deep Learning Assisted Efficient AdaBoost Algorithm for Breast Cancer Detection and Early Diagnosis. IEEE Access 8:96946–96954. https://doi.org/10.1109/ACCESS.2020.2993536

    Article  Google Scholar 

  11. Burt JR, Torosdagli N, Khosravan N et al (2018) Deep learning beyond cats and dogs: recent advances in diagnosing breast cancer with deep neural networks. Br J Radiol 91:20170545. https://doi.org/10.1259/bjr.20170545

    Article  Google Scholar 

  12. McKinney SM, Sieniek M, Godbole V et al (2020) International evaluation of an AI system for breast cancer screening. Nature 577:89–94. https://doi.org/10.1038/s41586-019-1799-6

    Article  Google Scholar 

  13. Ayana G, Dese K, Choe S (2021) Transfer Learning in Breast Cancer Diagnoses via Ultrasound Imaging. Cancers (Basel) 13:738. https://doi.org/10.3390/cancers13040738

    Article  Google Scholar 

  14. Ayana G, Park J, Jeong J-W, Choe S (2022) A Novel Multistage Transfer Learning for Ultrasound Breast Cancer Image Classification. Diagnostics 12:135. https://doi.org/10.3390/diagnostics12010135

    Article  Google Scholar 

  15. Ayana G, Park J, Choe S (2022) Patchless Multi-Stage Transfer Learning for Improved Mammographic Breast Mass Classification. Cancers (Basel) 14:1280. https://doi.org/10.3390/cancers14051280

    Article  Google Scholar 

  16. Wang X, Liang G, Zhang Y et al (2020) Inconsistent Performance of Deep Learning Models on Mammogram Classification. J Am Coll Radiol 17:796–803. https://doi.org/10.1016/j.jacr.2020.01.006

    Article  Google Scholar 

  17. Dese Gebremeskel K, Chung Kwa T, Hakkins Raj K et al (2021) Automatic Early Detection and Classification of Leukemia from Microscopic Blood Image. Abyssinia J Sci Technol 3:1–10. https://doi.org/10.20372/ajec.2021.v1.i1.160

    Article  Google Scholar 

  18. Dese K, Ayana G, Lamesgin Simegn G (2022) Low cost, non-invasive, and continuous vital signs monitoring device for pregnant women in low resource settings (Lvital device). HardwareX 11:e00276. https://doi.org/10.1016/j.ohx.2022.e00276

    Article  Google Scholar 

  19. Dese K, Raj H, Ayana G et al (2021) Accurate Machine-Learning-Based classification of Leukemia from Blood Smear Images. Clin Lymphoma Myeloma Leuk 21:e903–e914. https://doi.org/10.1016/j.clml.2021.06.025

    Article  Google Scholar 

  20. Lotter W, Diab AR, Haslam B et al (2021) Robust breast cancer detection in mammography and digital breast tomosynthesis using an annotation-efficient deep learning approach. Nat Med 27:244–249. https://doi.org/10.1038/s41591-020-01174-9

    Article  Google Scholar 

  21. Lotter W, Sorensen G, Cox D (2017) A multi-scale CNN and curriculum learning strategy for mammogram classification. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 10553 LNCS:169–177. https://doi.org/10.1007/978-3-319-67558-9_20

  22. Hosseini H, Xiao B, Jaiswal M, Poovendran R (2017) On the limitation of convolutional neural networks in recognizing negative images. Proc - 16th IEEE Int Conf Mach Learn Appl ICMLA 2017 2017-Decem:352–358. https://doi.org/10.1109/ICMLA.2017.0-136

  23. Chougrad H, Zouaki H, Alheyane O (2018) Deep Convolutional Neural Networks for breast cancer screening. Comput Methods Programs Biomed 157:19–30. https://doi.org/10.1016/j.cmpb.2018.01.011

    Article  Google Scholar 

  24. Li H, Niu J, Li D, Zhang C (2021) Classification of breast mass in two-view mammograms via deep learning. IET Image Process 15:454–467. https://doi.org/10.1049/ipr2.12035

    Article  Google Scholar 

  25. Agnes SA, Anitha J, Pandian SIA, Peter JD (2020) Classification of Mammogram Images Using Multiscale all Convolutional Neural Network (MA-CNN). J Med Syst 44:30. https://doi.org/10.1007/s10916-019-1494-z

    Article  Google Scholar 

  26. Xie L, Zhang L, Hu T et al (2020) Neural networks model based on an automated multi-scale method for mammogram classification. Knowledge-Based Syst 208:106465. https://doi.org/10.1016/j.knosys.2020.106465

    Article  Google Scholar 

  27. Carneiro G, Nascimento J, Bradley AP (2017) Automated Analysis of Unregistered Multi-View Mammograms with Deep Learning. IEEE Trans Med Imaging 36:2355–2365. https://doi.org/10.1109/TMI.2017.2751523

    Article  Google Scholar 

  28. Dhungel N, Carneiro G, Bradley AP (2017) A deep learning approach for the analysis of masses in mammograms with minimal user intervention. Med Image Anal 37:114–128. https://doi.org/10.1016/j.media.2017.01.009

    Article  Google Scholar 

  29. Huynh BQ, Li H, Giger ML (2016) Digital mammographic tumor classification using transfer learning from deep convolutional neural networks. J Med Imaging 3:034501. https://doi.org/10.1117/1.JMI.3.3.034501

    Article  Google Scholar 

  30. Kooi T, Gubern-Merida A, Mordang J-J, et al (2016) A Comparison Between a Deep Convolutional Neural Network and Radiologists for Classifying Regions of Interest in Mammography. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 9699:V–VI. https://doi.org/10.1007/978-3-319-41546-8

  31. Mudeng V, ChoeWoon S (2022) Deep neural network incorporating domain and resolution transformations model for histopathological image classification. Comput Electr Eng 104:108468. https://doi.org/10.1016/j.compeleceng.2022.108468

    Article  Google Scholar 

  32. Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

  33. Raghu M, Unterthiner T, Kornblith S, et al (2021) Do Vision Transformers See Like Convolutional Neural Networks?

  34. Mehta S, Lu X, Wu W et al (2022) End-to-End diagnosis of breast biopsy images with transformers. Med Image Anal 79:102466. https://doi.org/10.1016/j.media.2022.102466

    Article  Google Scholar 

  35. He Z, Lin M, Xu Z et al (2022) Deconv-transformer ( DecT ): A histopathological image classification model for breast cancer based on color deconvolution and transformer architecture. Inf Sci (Ny) 608:1093–1112. https://doi.org/10.1016/j.ins.2022.06.091

    Article  Google Scholar 

  36. Gheflati B, Rivaz H (2021) Vision Transformer for Classification of Breast Ultrasound Images

  37. Saidnassim N, Abdikenov B, Kelesbekov R, et al (2021) Self-supervised Visual Transformers for Breast Cancer Diagnosis. 2021 Asia-Pacific Signal Inf Process Assoc Annu Summit Conf APSIPA ASC 2021 - Proc 423–427

  38. Chen X, Zhang K, Abdoli N et al (2022) Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms. Diagnostics 12:1549. https://doi.org/10.3390/diagnostics12071549

    Article  Google Scholar 

  39. Ayana G, Dese K, Dereje Y et al (2023) Vision-Transformer-Based Transfer Learning for Mammogram Classification. Diagnostics 13:178. https://doi.org/10.3390/diagnostics13020178

    Article  Google Scholar 

  40. Ayana G, Choe S (2022) BUViTNet: Breast Ultrasound Detection via Vision Transformers. Diagnostics 12:2654. https://doi.org/10.3390/diagnostics12112654

    Article  Google Scholar 

  41. Heath M, Bowyer K, Kopans D, et al (2001) The Digital Database for Screening Mammography. In: Proceedings of the Fifth International Workshop on Digital Mammography. 212–218

  42. Heath M, Bowyer K, Kopans D et al (1998) Current Status of the Digital Database for Screening Mammography. Computational Imaging and Vision, vol 13. Springer, Dordrecht, pp 457–460

    Google Scholar 

  43. Moreira IC, Amaral I, Domingues I et al (2012) INbreast: Toward a Full-field Digital Mammographic Database. Acad Radiol 19:236–248. https://doi.org/10.1016/j.acra.2011.09.014

    Article  Google Scholar 

  44. Suckling J, Parker J, Dance D, et al (1994) The Mammographic Image Analysis Society Digital Mammogram Database Experta Medica. In: Int. Congr. Ser. 1069

  45. Rodrigues PS (2018) Breast Ultrasound Image. Mendeley Data. https://doi.org/10.17632/wmy84gzngw.1

  46. Al-Dhabyani W, Gomaa M, Khaled H, Fahmy A (2020) Dataset of breast ultrasound images. Data Br 28:104863. https://doi.org/10.1016/j.dib.2019.104863

    Article  Google Scholar 

  47. Tougui I, Jilbab A, El MJ (2021) Impact of the choice of cross-validation techniques on the results of machine learning-based diagnostic applications. Healthc Inform Res 27:189–199. https://doi.org/10.4258/HIR.2021.27.3.189

    Article  Google Scholar 

Download references

Funding

This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023–00240521) and project for Industry-University-Research Institute platform cooperation R&D funded by Korea Ministry of SMEs and Startups in 2022 (S3310765).

Author information

Authors and Affiliations

  1. Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi, 39253, Korea

    Gelan Ayana & Se-woon Choe

  2. School of Biomedical Engineering, Jimma University, 378, Jimma, Ethiopia

    Gelan Ayana

  3. Department of IT Convergence Engineering, Kumoh National Institute of Technology, Gumi, 39253, Korea

    Se-woon Choe

Authors
  1. Gelan Ayana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Se-woon Choe
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Gelan Ayana and Se-woon Choe; Methodology: Gelan Ayana and Se-woon Choe; Formal analysis and investigation: Gelan Ayana and Se-woon Choe; Writing—original draft preparation: Gelan Ayana and Se-woon Choe; Writing—review and editing: Gelan Ayana and Se-woon Choe; Funding acquisition: Se-woon Choe; Resources: Se-woon Choe; Supervision: Se-woon Choe.

Corresponding author

Correspondence to Se-woon Choe.

Ethics declarations

Ethical Approval

This work does not include any studies with human participants or animals performed by any of the authors.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 20 KB)

Supplementary file2 (DOCX 950 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ayana, G., Choe, Sw. Vision Transformers-Based Transfer Learning for Breast Mass Classification From Multiple Diagnostic Modalities. J. Electr. Eng. Technol. (2024). https://doi.org/10.1007/s42835-024-01904-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42835-024-01904-w

Keywords

Search

Navigation