Skip to main content

Advertisement

SpringerLink
Log in

Multimodal classification of breast cancer using feature level fusion of mammogram and ultrasound images in machine learning paradigm

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The manual examination of Breast Cancer (BC) images for disease detection is prone to error, is time-consuming, and has low accuracy. The Computer-Aided Detection (CAD) system can overcome the above-mentioned limitations, however, the existing single modality studies have limited clinical applications because the Radiologists mostly make use of both ultrasound with a corresponding mammogram and vice-versa for final diagnosis. This paper presents a novel semi-automated multimodal classification system of breast tumors by combining features from both mammogram and ultrasound images. Forty-two grayscale features were extracted followed by statistical significance analysis to determine the most relevant features and classify the tumors as benign or malignant. A new database consisting of 43 mammograms and 43 ultrasounds is used in the experiments. The size of the dataset is increased further by applying several data augmentation techniques. Also, filtering is applied to deal with artifacts and noise present in the images which may affect the performance accuracy of a model. The performance of the proposed multimodal approach is evaluated using different machine learning classifiers under a tenfold data division protocol. The results demonstrate that combined mammogram and ultrasound achieve a high classification accuracy of 98.84% with cubic Support Vector Machine (SVM) for the proposed multimodal CAD system compared to mammography and ultrasound alone which achieves an accuracy of 93.41% and 91.67% respectively. Our results support the current clinical practice of utilizing mammogram and ultrasound together for improved BC screening.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

Data is not available with this manuscript but can be made available on reasonable request after permission of IEC.

References

  1. Al-Antari MA, Al-Masni MA, Choi MT, Han SM, Kim TS (2018) A fully integrated computer-aided diagnosis system for digital X-ray mammograms via deep learning detection, segmentation, and classification. Int J Med Inform 117:44–54. https://doi.org/10.1016/j.ijmedinf.2018.06.003

    Article  PubMed  Google Scholar 

  2. American Cancer Society (2022) Cancer Facts & Figures 2022. https://www.cancer.org/content/dam/cancerorg/research/cancer-facts-and-statistics/annual-cancer-facts-and-figures/2022/2022-cancer-facts-and-figures.pdf. Accessed 02 June 2022

  3. Arya N, Saha S (2021) Generative Incomplete Multi-View Prognosis Predictor for Breast Cancer: GIMPP. IEEE/ACM Trans. Comput Biol Bioinform. https://doi.org/10.1109/TCBB.2021.3090458

  4. Atrey K, Sharma Y, Bodhey NK, Singh BK (2019) Breast cancer prediction using dominance-based feature filtering approach: A comparative investigation in machine learning archetype. Braz Arch Biol Technol 62:e19180486

    Article  Google Scholar 

  5. Bhesdadiya R, Kalola J, Jethva M (2020) Mammographic and Ultrasonographic Evaluation of Breast Lesions with Pathological Correlation. Int J Contemp Med Surg Radiol 5(1):A158–A161. https://doi.org/10.21276/ijcmsr.2020.5.1.35

    Article  Google Scholar 

  6. Byra M (2018) Discriminant analysis of neural style representations for breast lesion classification in ultrasound. Biocybern Biomed Eng 38(3):684–690. https://doi.org/10.1016/j.bbe.2018.05.003

    Article  Google Scholar 

  7. Chiao JY, Chen KY, Liao KY, Hsieh PH, Zhang G, Huang TC (2019) Detection and classification the breast tumors using mask R-CNN on sonograms. Medicine 98(19):e15200. https://doi.org/10.1097/MD.0000000000015200

    Article  PubMed  PubMed Central  Google Scholar 

  8. Cong J, Wei B, He Y, Yin Y, Zheng Y (2017) A selective ensemble classification method combining mammography images with ultrasound images for breast cancer diagnosis. Comput Math Methods Med 2017:1–7. https://doi.org/10.1155/2017/4896386

    Article  ADS  Google Scholar 

  9. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297. https://doi.org/10.1007/BF00994018

    Article  Google Scholar 

  10. Daoud MI, Atallah AA, Awwad F, Al-Najjar M, Alazrai R (2019) Automatic superpixel-based segmentation method for breast ultrasound images. Expert Syst Appl 121:78–96. https://doi.org/10.1016/j.eswa.2018.11.024

    Article  Google Scholar 

  11. Devarakonda M, Lingamgunta S (2019) Triple-modality breast cancer diagnosis and analysis in middle aged women by logistic regression. Int J Innov Technol Explor Eng 8(4):555–562

    Google Scholar 

  12. Fang H, Fan H, Lin S, Qing Z, Sheykhahmad FR (2021) Automatic breast cancer detection based on optimized neural network using whale optimization algorithm. Int J Imaging Syst Technol 31(1):425–438. https://doi.org/10.1002/ima.22468

    Article  Google Scholar 

  13. Gu K, Tao D, Qiao JF, Lin W (2017) Learning a no-reference quality assessment model of enhanced images with big data. IEEE Trans Neural Netw Learn Syst 29(4):1301–1313. https://doi.org/10.1109/TNNLS.2017.2649101

    Article  PubMed  Google Scholar 

  14. Gu K, Zhai G, Lin W, Liu M (2015) The analysis of image contrast: From quality assessment to automatic enhancement. IEEE Trans Cybern 46(1):284–297. https://doi.org/10.1109/TCYB.2015.2401732

    Article  PubMed  Google Scholar 

  15. Gu K, Zhai G, Lin W, Yang X, Zhang W (2015) No-reference image sharpness assessment in autoregressive parameter space. IEEE Trans Image Process 24(10):3218–3231. https://doi.org/10.1109/TIP.2015.2439035

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  16. Gu K, Zhai G, Yang X, Zhang W, Chen CW (2014) Automatic contrast enhancement technology with saliency preservation. IEEE Trans Circuits Syst Video Technol 25(9):1480–1494. https://doi.org/10.1109/TCSVT.2014.2372392

    Article  Google Scholar 

  17. Habib G, Kiryati N, Sklair-Levy M et al (2020) Automatic breast lesion classification by joint neural analysis of mammography and ultrasound. In Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures, Springer, Cham pp. 125–135. https://doi.org/10.1007/978-3-030-60946-7_13

  18. Hamouda SK, El-Ezz RH, Wahed ME (2017) Enhancement accuracy of breast tumor diagnosis in digital mammograms. J Biomed Sci 6(4):28. https://doi.org/10.4172/2254-609X.100072

    Article  Google Scholar 

  19. Jabeen K, Khan MA, Alhaisoni M, Tariq U, Zhang YD, Hamza A, Mickus A, Damaševičius R (2022) Breast cancer classification from ultrasound images using probability-based optimal deep learning feature fusion. Sensors 22(3):807. https://doi.org/10.3390/s22030807

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  20. Kaur P, Singh G, Kaur P (2019) Intellectual detection and validation of automated mammogram breast cancer images by multi-class SVM using deep learning classification. Inform Med Unlocked 16:100151. https://doi.org/10.1016/j.imu.2019.01.001

    Article  Google Scholar 

  21. Kriti VJ, Agarwal R (2019) Effect of despeckle filtering on classification of breast tumors using ultrasound images. Biocybern Biomed Eng 39(2):536–560. https://doi.org/10.1016/j.bbe.2019.02.004

    Article  Google Scholar 

  22. Kriti VJ, Agarwal R (2020) Deep feature extraction and classification of breast ultrasound images. Multimed Tools Appl 79(37):27257–27292. https://doi.org/10.1007/s11042-020-09337-z

    Article  Google Scholar 

  23. Lee J (2017) Practical and illustrated summary of updated BI-RADS for ultrasonography. Ultrasonography 36(1):71. https://doi.org/10.14366/usg.16034

    Article  PubMed  Google Scholar 

  24. Melekoodappattu JG, Subbian PS, Queen MF (2021) Detection and classification of breast cancer from digital mammograms using hybrid extreme learning machine classifier. Int J Imaging Syst Technol 31(2):909–920. https://doi.org/10.1002/ima.22484

    Article  Google Scholar 

  25. Mohammed MA, Al-Khateeb B, Rashid AN, Ibrahim DA, Abd Ghani MK, Mostafa SA (2018) Neural network and multi-fractal dimension features for breast cancer classification from ultrasound images. Comput Electr Eng 70:871–882

    Article  Google Scholar 

  26. Moon WK, Lee YW, Ke HH et al (2020) Computer-aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks. Comput Methods Programs Biomed 190:105361. https://doi.org/10.1016/j.cmpb.2020.105361

    Article  PubMed  Google Scholar 

  27. Oyelade ON, Ezugwu AE (2021) A deep learning model using data augmentation for detection of architectural distortion in whole and patches of images. Biomed Signal Process Control 65:102366. https://doi.org/10.1016/j.bspc.2020.102366

    Article  Google Scholar 

  28. Pawar MM, Talbar SN (2016) Genetic fuzzy system (GFS) based wavelet co-occurrence feature selection in mammogram classification for breast cancer diagnosis. Perspect Sci 8:247–250. https://doi.org/10.1016/j.pisc.2016.04.042

    Article  Google Scholar 

  29. Du Prel JB, Röhrig B, Hommel G, Blettner M (2010) Choosing statistical tests: part 12 of a series on evaluation of scientific publications. Dtsch Arztebl Int 107(19):343–348. https://doi.org/10.3238/arztebl.2010.0343

    Article  PubMed  PubMed Central  Google Scholar 

  30. Prochazka A, Gulati S, Holinka S, Smutek D (2019) Classification of thyroid nodules in ultrasound images using direction-independent features extracted by two-threshold binary decomposition. Technol Cancer Res Treat 18:1533033819830748. https://doi.org/10.1177/1533033819830748

    Article  PubMed  PubMed Central  Google Scholar 

  31. Rahman TY, Mahanta LB, Choudhury H, Das AK, Sarma JD (2020) Study of morphological and textural features for classification of oral squamous cell carcinoma by traditional machine learning techniques. Cancer Rep 3(6):e1293. https://doi.org/10.1002/cnr2.1293

    Article  Google Scholar 

  32. Raj JR, Rahman SM, Anand S (2016) Computational assessment of breast tumour differentiation using multimodal data. Inform Med Unlocked 2:70–77. https://doi.org/10.1016/j.imu.2016.04.001

    Article  Google Scholar 

  33. Rao AA, Feneis J, Lalonde C, Ojeda-Fournier H (2016) A pictorial review of changes in the BI-RADS fifth edition. Radiographics 36(3):623–639. https://doi.org/10.1148/rg.2016150178

    Article  PubMed  Google Scholar 

  34. Rouhi R, Jafari M, Kasaei S, Keshavarzian P (2015) Benign and malignant breast tumors classification based on region growing and CNN segmentation. Expert Syst Appl 42(3):990–1002. https://doi.org/10.1016/j.eswa.2014.09.020

    Article  Google Scholar 

  35. Saranyaraj D, Manikandan M, Maheswari S (2020) A deep convolutional neural network for the early detection of breast carcinoma with respect to hyper-parameter tuning. Multimed Tools Appl 79(15–16):11013–11038. https://doi.org/10.1007/s11042-018-6560-x

    Article  Google Scholar 

  36. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6(1):1–48. https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

  37. Sidiropoulos KP, Kostopoulos SA, Glotsos DT et al (2013) Multimodality GPU-based computer-assisted diagnosis of breast cancer using ultrasound and digital mammography images. Int J Comput Assist Radiol Surg 8(4):547–560. https://doi.org/10.1007/s11548-013-0813-y

    Article  PubMed  Google Scholar 

  38. Singh BK (2019) Determining relevant biomarkers for prediction of breast cancer using anthropometric and clinical features: A comparative investigation in machine learning paradigm. Biocybern Biomed Eng 39(2):393–409. https://doi.org/10.1016/j.bbe.2019.03.001

    Article  Google Scholar 

  39. Singh BK, Jain P, Banchhor SK, Verma K (2019) Performance evaluation of breast lesion detection systems with expert delineations: a comparative investigation on mammographic images. Multimed Tools Appl 78(16):22421–22444. https://doi.org/10.1007/s11042-019-7570-z

    Article  Google Scholar 

  40. Singh BK, Verma K, Panigrahi L, Thoke AS (2017) Integrating radiologist feedback with computer aided diagnostic systems for breast cancer risk prediction in ultrasonic images: An experimental investigation in machine learning paradigm. Expert Syst Appl 90:209–223. https://doi.org/10.1016/j.eswa.2017.08.020

    Article  Google Scholar 

  41. Singh BK, Verma K, Thoke AS (2015) Investigations on impact of feature normalization techniques on classifier’s performance in breast tumor classification. Int J Comput Appl 116(19):11–15

    Google Scholar 

  42. Singh BK, Verma K, Thoke AS (2016) Fuzzy cluster based neural network classifier for classifying breast tumors in ultrasound images. Expert Syst Appl 66:114–123. https://doi.org/10.1016/j.eswa.2016.09.006

    Article  Google Scholar 

  43. Singh BK, Verma K, Thoke AS, Suri JS (2017) Risk stratification of 2D ultrasound-based breast lesions using hybrid feature selection in machine learning paradigm. Measurement 105:146–157. https://doi.org/10.1016/j.measurement.2017.01.016

    Article  ADS  Google Scholar 

  44. Sun D, Wang M, Li A (2018) A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data. IEEE/ACM Trans Comput Biol Bioinform 16(3):841–850. https://doi.org/10.1109/TCBB.2018.2806438

    Article  Google Scholar 

  45. Sung H, Ferlay J, Siegel RL et al (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA A Cancer J Clin 71(3):209–249. https://doi.org/10.3322/caac.21660

    Article  Google Scholar 

  46. Vidivelli S, Devi SS (2023) Breast cancer detection model using fuzzy entropy segmentation and ensemble classification. Biomed Signal Process Control 80:104236. https://doi.org/10.1016/j.bspc.2022.104236

    Article  Google Scholar 

  47. Vijayarajeswari R, Parthasarathy P, Vivekanandan S, Basha AA (2019) Classification of mammogram for early detection of breast cancer using SVM classifier and Hough transform. Measurement 146:800–805. https://doi.org/10.1016/j.measurement.2019.05.083

    Article  ADS  Google Scholar 

  48. Xu SS, Chang CC, Su CT, Phu PQ (2019) Classification of liver diseases based on ultrasound image texture features. Appl Sci 9(2):342. https://doi.org/10.3390/app9020342

    Article  ADS  Google Scholar 

  49. Yi A, Jang MJ, Yim D, Kwon BR, Shin SU, Chang JM (2021) Addition of screening breast US to digital mammography and digital breast Tomosynthesis for breast cancer screening in women at average risk. Radiology 298(3):568–575. https://doi.org/10.1148/radiol.2021203134

    Article  PubMed  Google Scholar 

  50. Yuan Y, Giger ML, Li H, Bhooshan N, Sennett CA (2012) Correlative analysis of FFDM and DCE-MRI for improved breast CADx. J Med Biol Eng 32(1):42–50

    Article  Google Scholar 

  51. Zhang X, Li H, Wang C et al (2021) Evaluating the accuracy of breast cancer and molecular subtype diagnosis by ultrasound image deep learning model. Front Oncol 11:623506. https://doi.org/10.3389/fonc.2021.623506

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Zhang YD, Satapathy SC, Guttery DS, Górriz JM, Wang SH (2021) Improved breast cancer classification through combining graph convolutional network and convolutional neural network. Inf Process Manag 58(2):102439. https://doi.org/10.1016/j.ipm.2020.102439

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, National Institute of Technology Raipur, Raipur, Chhattisgarh, 492010, India

    Kushangi Atrey & Bikesh Kumar Singh

  2. Department of Radiodiagnosis, All India Institute of Medical Sciences Raipur, Raipur, Chhattisgarh, 492099, India

    Narendra Kuber Bodhey

Authors
  1. Kushangi Atrey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bikesh Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Narendra Kuber Bodhey
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Kushangi Atrey: Methodology, Software, Writing-original draft. Bikesh Kumar Singh: Conceptualization, Methodology, Supervision. Narendra Kuber Bodhey: Conceptualization, Data collection and development, Supervision. All authors reviewed the manuscript.

Corresponding author

Correspondence to Bikesh Kumar Singh.

Ethics declarations

Ethical approval and Consent to participate

The study was approved by the Institutional ethics committee (IEC), NIT Raipur letter no.: NITRR/IEC/2019/04. Informed consent was obtained from all the participants in this study.

Human and animal ethics

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Atrey, K., Singh, B.K. & Bodhey, N.K. Multimodal classification of breast cancer using feature level fusion of mammogram and ultrasound images in machine learning paradigm. Multimed Tools Appl 83, 21347–21368 (2024). https://doi.org/10.1007/s11042-023-16414-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16414-6

Keywords

Search

Navigation