Abstract
Purpose
Skin cancer is one of the ten most common cancer types in the world. Early diagnosis and treatment can effectively reduce the mortality of patients. Therefore, it is of great significance to develop an intelligent diagnosis system for skin cancer. According to the survey, at present, most intelligent diagnosis systems of skin cancer only use skin image data, but the multi-modal cross-fusion analysis using image data and patient clinical data is limited. Therefore, to further explore the complementary relationship between image data and patient clinical data, we propose multimode data fusion diagnosis network (MDFNet), a framework for skin cancer based on data fusion strategy.
Methods
MDFNet establishes an effective mapping among heterogeneous data features, effectively fuses clinical skin images and patient clinical data, and effectively solves the problems of feature paucity and insufficient feature richness that only use single-mode data.
Results
The experimental results present that our proposed smart skin cancer diagnosis model has an accuracy of 80.42%, which is an improvement of about 9% compared with the model accuracy using only medical images, thus effectively confirming the unique fusion advantages exhibited by MDFNet.
Conclusions
This illustrates that MDFNet can not only be applied as an effective auxiliary diagnostic tool for skin cancer diagnosis, help physicians improve clinical decision-making ability and effectively improve the efficiency of clinical medicine diagnosis, but also its proposed data fusion method fully exerts the advantage of information convergence and has a certain reference value for the intelligent diagnosis of numerous clinical diseases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alizadeh SM, Mahloojifar A (2021) Automatic skin cancer detection in dermoscopy images by combining convolutional neural networks and texture features. Int J Imaging Syst Technol 31(2):695–707. https://doi.org/10.1002/ima.22490
Al-Masni MA, Kim D-H, Kim T-S (2020) Multiple skin lesions diagnostics via integrated deep convolutional networks for segmentation and classification. Comput Methods Programs Biomed 190:105351. https://doi.org/10.1016/j.cmpb.2020.105351
Antropova N, Huynh BQ, Giger ML (2017) A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets. Med Phys 44(10):5162–5171. https://doi.org/10.1002/mp.12453
Bianco S, Cadene R, Celona L, Napoletano P (2018) Benchmark analysis of representative deep neural network architectures. IEEE Access 6:64270–64277
Cancer Facts & Figures 2022. (2022). https://www.cancer.org/research/cancer-facts-statistics/all-cancer-factsfigures/cancer-facts-figures-2022.html. Accessed 10 Mar 2022
Canziani, A., Paszke, A., & Culurciello, E. (2016). An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678
Celebi ME, Codella N, Halpern A (2019) Dermoscopy image analysis: overview and future directions. IEEE J Biomed Health Inform 23(2):474–478. https://doi.org/10.1109/JBHI.2019.2895803
Chen C, Chen F, Yang B, Zhang K, Lv X, Chen C (2022) A novel diagnostic method: FT-IR, Raman and derivative spectroscopy fusion technology for the rapid diagnosis of renal cell carcinoma serum. Spectrochim Acta Part A Mol Biomol Spectrosc 269:120684. https://doi.org/10.1016/j.saa.2021.120684
Corona R, Dogliotti E, D’Errico M, Sera F, Iavarone I, Baliva G et al (2001) Risk factors for basal cell carcinoma in a mediterranean population: role of recreational sun exposure early in life. Arch Dermatol 137(9):1162–1168. https://doi.org/10.1001/archderm.137.9.1162
Dai C, Liu X, Lai J (2020) Human action recognition using two-stream attention based LSTM networks. Appl Soft Comput 86:105820. https://doi.org/10.1016/j.asoc.2019.105820
Depeursingea A, Racoceanub D, Iavindrasanaa J, Cohena G, Platonc A, Polettic PA, Müllera H (2010) Fusing visual and clinical information for lung tissue classification in HRCT data. Artif Intell Med. https://doi.org/10.1016/j.artmed.2010.04.006
Ferlay J, Colombet M, Soerjomataram I, Parkin DM, Piñeros M, Znaor A, Bray F (2021) Cancer statistics for the year 2020: an overview. Int J Cancer 149(4):778–789
Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H. (2019). Dual attention network for scene segmentation. Paper presented at the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
Gong A, Yao X, Lin W (2020) Classification for dermoscopy images using convolutional neural networks based on the ensemble of individual advantage and group decision. IEEE Access 8:155337–155351. https://doi.org/10.1109/ACCESS.2020.3019210
Goyal M, Oakley A, Bansal P, Dancey D, Yap MH (2019) Skin lesion segmentation in dermoscopic images with ensemble deep learning methods. IEEE Access 8:4171–4418. https://doi.org/10.1109/ACCESS.2019.2960504
Guo S, Yang Z (2018) Multi-channel-ResNet: an integration framework towards skin lesion analysis. Inform Med Unlocked 12:67–74. https://doi.org/10.1016/j.imu.2018.06.006
He K, Zhang X, Ren S, Sun J. (2016). Deep residual learning for image recognition. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hu J, Shen L, Sun G. (2018). Squeeze-and-excitation networks. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017). Densely connected convolutional networks. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
Jones MS, Jones PC, Stern SL, Elashoff D, Hoon DS, Thompson J et al (2017) The impact of smoking on sentinel node metastasis of primary cutaneous melanoma. Ann Surg Oncol 24(8):2089–2094. https://doi.org/10.1245/s10434-017-5775-9
Khan IU, Aslam N, Anwar T, Aljameel SS, Ullah M, Khan R et al (2021) Remote diagnosis and triaging model for skin cancer using efficientnet and extreme gradient boosting. Complexity. https://doi.org/10.1155/2021/5591614
Li W, Zhuang J, Wang R, Zhang J, Zheng W-S (2020). Fusing metadata and dermoscopy images for skin disease diagnosis. Paper presented at the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Doi: https://doi.org/10.1109/ISBI45749.2020.9098645
Liu Y, Jain A, Eng C, Way DH, Lee K, Bui P et al (2020) A deep learning system for differential diagnosis of skin diseases. Nat Med 26(6):900–908
Long GV, Menzies AM, Nagrial AM, Haydu LE, Hamilton AL, Mann GJ et al (2011) Prognostic and clinicopathologic associations of oncogenic BRAF in metastatic melanoma. J Clin Oncol 29(10):1239–1246
Mahbod A, Schaefer G, Wang C, Dorffner G, Ecker R, Ellinger I (2020) Transfer learning using a multi-scale and multi-network ensemble for skin lesion classification. Comput Methods Programs Biomed 193:105475. https://doi.org/10.1016/j.cmpb.2020.105475
Nachbar F, Stolz W, Merkle T, Cognetta AB, Vogt T, Landthaler M et al (1994) The ABCD rule of dermatoscopy: high prospective value in the diagnosis of doubtful melanocytic skin lesions. J Am Acad Dermatol 30(4):551–559. https://doi.org/10.1016/S0190-9622(94)70061-3
Niu Z, Zhong G, Yu H (2021) A review on the attention mechanism of deep learning. Neurocomputing 452:48–62. https://doi.org/10.1016/j.neucom.2021.03.091
Odenbro Å, Bellocco R, Boffetta P, Lindelöf B, Adami J (2005) Tobacco smoking, snuff dipping and the risk of cutaneous squamous cell carcinoma: a nationwide cohort study in Sweden. Br J Cancer 92(7):1326–1328. https://doi.org/10.1016/j.cmpb.2020.105475
Pacheco AG, Krohling RA (2020) The impact of patient clinical information on automated skin cancer detection. Comput Biol Med 116:103545. https://doi.org/10.1016/j.compbiomed.2019.103545
Pacheco AG, Krohling RA (2021) An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification. IEEE J Biomed Health Inform 25(9):3554–3563. https://doi.org/10.1109/JBHI.2021.3062002
Pacheco AG, Lima GR, Salomão AS, Krohling B, Biral IP, de Angelo GG et al (2020) PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones. Data Brief 32:106221. https://doi.org/10.1016/j.dib.2020.106221
Ramachandram D, Taylor GW (2017) Deep multimodal learning: A survey on recent advances and trends. IEEE Signal Process Mag 34(6):96–108. https://doi.org/10.1109/MSP.2017.2738401
Reis HC, Turk V, Khoshelham K, Kaya S (2022) InSiNet: a deep convolutional approach to skin cancer detection and segmentation. Med Biol Eng Comput 60:1–20
Sawada Y, Gallo RL (2021) Role of Epigenetics in the Regulation of Immune Functions of the Skin. J Investig Dermatol 141(5):1157–1166
Sawada Y, Nakamura M (2021) Daily lifestyle and cutaneous malignancies. Int J Mol Sci 22(10):5227. https://doi.org/10.3390/ijms22105227
Schlemper, J., Oktay, O., Chen, L., Matthew, J., Knight, C., Kainz, B., . . . Rueckert, D. (2018). Attention-gated networks for improving ultrasound scan plane detection. arXiv preprint arXiv:1804.05338. https://doi.org/10.48550/arXiv.1804.05338
Sedghi A, Mehrtash A, Jamzad A, Amalou A, Wells WM, Kapur T et al (2020) Improving detection of prostate cancer foci via information fusion of MRI and temporal enhanced ultrasound. Int J Comput Assist Radiol Surg 15(7):1215–1223. https://doi.org/10.1007/s11548-020-02172-5
Sharma, S., Kiros, R., & Salakhutdinov, R. (2015). Action recognition using visual attention. arXiv preprint arXiv:1511.04119. https://doi.org/10.48550/arXiv.1511.04119
Siddiqui SY, Naseer I, Khan MA, Mushtaq MF, Naqvi RA, Hussain D, Haider A (2021) Intelligent breast cancer prediction empowered with fusion and deep learning. Comput Mater Contin. https://doi.org/10.32604/cmc.2021.013952
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Sun D, Li A, Tang B, Wang M (2018) Integrating genomic data and pathological images to effectively predict breast cancer clinical outcome. Comput Methods Programs Biomed 161:45–53. https://doi.org/10.1016/j.cmpb.2018.04.008
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016). Rethinking the inception architecture for computer vision. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
Watson M, Holman DM, Maguire-Eisen M (2016) Ultraviolet radiation exposure and its impact on skin cancer risk. Semin Oncol Nurs. https://doi.org/10.1016/j.soncn.2016.05.005
Wei H, Jafarian A, Zeidman P, Litvak V, Razi A, Hu D, Friston KJ (2020) Bayesian fusion and multimodal DCM for EEG and fMRI. Neuroimage 211:116595. https://doi.org/10.1016/j.neuroimage.2020.116595
Wojna, Z., Gorban, A. N., Lee, D.-S., Murphy, K., Yu, Q., Li, Y., & Ibarz, J. (2017). Attention-based extraction of structured information from street view imagery. Paper presented at the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). https://doi.org/10.1109/ICDAR.2017.143
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018). Cbam: Convolutional block attention module. Paper presented at the Proceedings of the European conference on computer vision (ECCV).
Yan C, Tu Y, Wang X, Zhang Y, Hao X, Zhang Y, Dai Q (2019) STAT: spatial-temporal attention mechanism for video captioning. IEEE Trans Multimedia 22(1):229–241. https://doi.org/10.1109/TMM.2019.2924576
Yue F, Chen C, Yan Z, Chen C, Guo Z, Zhang Z et al (2020) Fourier transform infrared spectroscopy combined with deep learning and data enhancement for quick diagnosis of abnormal thyroid function. Photodiagn Photodyn Ther. https://doi.org/10.1016/j.pdpdt.2020.101923
Zhang Y-D, Dong Z, Wang S-H, Yu X, Yao X, Zhou Q et al (2020) Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation. Information Fusion 64:149–187. https://doi.org/10.1016/j.inffus.2020.07.006
Zhao, H., Zhang, Y., Liu, S., Shi, J., Loy, C. C., Lin, D., & Jia, J. (2018). Psanet: Point-wise spatial attention network for scene parsing. Paper presented at the Proceedings of the European conference on computer vision (ECCV).
Zhu W, Wang X, Li H (2019) Multi-modal deep analysis for multimedia. IEEE Trans Circuits Syst Video Technol 30(10):3740–3764. https://doi.org/10.1109/TCSVT.2019.2940647
Acknowledgements
This work is supported by the Distinguished Young Talents Project of Natural Science Foundation of Xinjiang Uygur Autonomous Region (2022D01E11 and 2022D01E27), the project of scientific and technological assistance to Xinjiang (No.2020E0269).
Funding
This work was funded by the Distinguished Young Talents Project of Natural Science Foundation of Xinjiang Uygur Autonomous Region (Grant number 2022D01E11 and 2022D01E27), the project of scientific and technological assistance to Xinjiang (Grant number No.2020E0269). Xiaoyi Lv declared that the funds were supported by the Distinguished Young Talents Project of Natural Science Foundation of Xinjiang Uygur Autonomous Region (Grant number 2022D01E11 and 2022D01E27) and the project of scientific and technological assistance to Xinjiang (Grant number No.2020E0269).
Author information
Authors and Affiliations
Contributions
Qian Chen is the experimental designer and executor of this research, completing data analysis and writing the first draft of the paper; Min Li guides experimental analysis and assists in thesis writing and revision; Chen Chen, Panyun Zhou participate in the experimental design and analysis of experimental results;Xiaoyi Lv is the designer and person in charge of the project. Cheng Chen guides the experimental design, data analysis and. All authors read and agreed to the final text.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflicting financial interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chen, Q., Li, M., Chen, C. et al. MDFNet: application of multimodal fusion method based on skin image and clinical data to skin cancer classification. J Cancer Res Clin Oncol 149, 3287–3299 (2023). https://doi.org/10.1007/s00432-022-04180-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00432-022-04180-1