Abstract
We present a study on the fusion of pixel data and patient metadata (age, gender, and body location) for improving the classification of skin lesion images. The experiments have been conducted with the ISIC 2019 skin lesion classification challenge data set. Taking two plain convolutional neural networks (CNNs) as a baseline, metadata are merged using either non-neural machine learning methods (tree-based and support vector machines) or shallow neural networks. Results show that shallow neural networks outperform other approaches in all overall evaluation measures. However, despite the increase in the classification accuracy (up to +19.1%), interestingly, the average per-class sensitivity decreases in three out of four cases for CNNs, thus suggesting that using metadata penalizes the prediction accuracy for lower represented classes. A study on the patient metadata shows that age is the most useful metadatum as a decision criterion, followed by body location and gender.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Argenziano, G., et al.: Seven-point checklist of dermoscopy revisited: seven-point checklist of dermoscopy revisited. Br. J. Dermatol. 164(4), 785–790 (2011)
Emre Celebi, M., Codella, N., Halpern, A.: Dermoscopy image analysis: overview and future directions. IEEE J. Biomed. Health Inf. 23(2), 474–478 (2019)
Codella, N.C.F., et al.: Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC). arXiv:1710.05006 [cs], October 2017
Combalia, M., et al.: BCN20000: Dermoscopic Lesions in the Wild. arXiv:1908.02288 [cs, eess], August 2019
Curiel-Lewandrowski, C., et al.: Artificial intelligence approach in melanoma. In: Fisher, D.E., Bastian, B.C. (eds.) Melanoma, pp. 1–31. Springer, New York (2019)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE, Miami, June 2009
Hospital Clínic de Barcelona Department of Dermatology. Bcn\_20000 dataset
Donahue, J., et al.: DeCAF: a deep convolutional activation feature for generic visual recognition. In: Xing, E.P., Jebara, T. (eds.) Proceedings of the 31st International Conference on Machine Learning. Proceedings of Machine Learning Research, Bejing, China, vol. 32, pp. 647–655, June 2014. PMLR
Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115 (2017)
Fujisawa, Y., et al.: Deep-learning-based, computer-aided classifier developed with a small dataset of clinical images surpasses board-certified dermatologists in skin tumour diagnosis. Br. J. Dermatol. 180(2), 373–381 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Kawahara, J., Daneshvar, S., Argenziano, G., Hamarneh, G.: Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE J. Biomed. Health Inf. 23(2), 538–546 (2019)
Kawahara J., Hamarneh, G.: Visual diagnosis of dermatological disorders: human and machine performance. arXiv:1906.01256 [cs], June 2019
Komura, T., Holden, D., Saito, J.: Phase-functioned neural networks for character control. ACM Trans. Graph. 36(4), 1–13 (2017). Siggraph 2017; Conference date: 30–07-2017 Through 03–08-2017
Mishra, N.K., Emre Celebi, M.: An Overview of Melanoma Detection in Dermoscopy Images Using Image Processing and Machine Learning. arXiv:1601.07843 [cs, stat], January 2016
Nunnari, F., Sonntag, D.: A CNN toolbox for skin cancer classification (2019). https://arxiv.org/abs/1908.08187. DFKI Technical report
Pacheco, A.G.C., Krohling, R.A.: The impact of patient clinical information on automated skin cancer detection. arXiv:1909.12912 [cs, eess, stat], September 2019
Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2014
Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs], September 2014
Sonntag, D., Nunnari, F., Profitlich, H.: The Skincare project, an interactive deep learning system for differential diagnosis of malignant skin lesions. Technical report (2020). https://arxiv.org/abs/2005.09448
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., Liu, C.: A survey on deep transfer learning. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 270–279. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_27
Medical University of Vienna ViDIR Group, Department of Dermatology. Ham10000 dataset. https://doi.org/10.1038/sdata.2018.161
Yap, J., Yolland, W., Tschandl, P.: Multimodal skin lesion classification using deep learning. Exp. Dermatol. 27(11), 1261–1267 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 IFIP International Federation for Information Processing
About this paper
Cite this paper
Nunnari, F., Bhuvaneshwara, C., Ezema, A.O., Sonntag, D. (2020). A Study on the Fusion of Pixels and Patient Metadata in CNN-Based Classification of Skin Lesion Images. In: Holzinger, A., Kieseberg, P., Tjoa, A., Weippl, E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2020. Lecture Notes in Computer Science(), vol 12279. Springer, Cham. https://doi.org/10.1007/978-3-030-57321-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-57321-8_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-57320-1
Online ISBN: 978-3-030-57321-8
eBook Packages: Computer ScienceComputer Science (R0)
