Skip to main content
SpringerLink
Log in

Tackling the class imbalanced dermoscopic image classification using data augmentation and GAN

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Dermoscopy is a noninvasive way to examine and diagnose skin lesions, e.g. nevus and melanoma, and is a critical step for skin cancer detection. Accurate classification of dermoscopic images can detect skin cancer at an early stage and bring social and economic impact to patients and communities. Using deep learning methods to classify dermoscopic images has shown superior performance, but existing research often overlooks the class imbalance in the data. In addition, although a handful of public datasets are available for skin cancer research, these datasets are generally not large enough for deep learning algorithms to produce accurate results. In this paper, we propose to use data augmentation and generative adversarial networks (GAN) to tackle class-imbalanced dermoscopic image classification. Our main objectives are to determine (1) how state-of-the-art fine-tuned deep learning models perform on class-imbalanced dermoscopic images, (2) whether data augmentation and GAN can help alleviate class imbalances to improve classification accuracy, and (3) which method is more effective in addressing the class imbalance. By using public datasets and a carefully designed framework to generate augmented images and synthetic images, our research provides clear answers to these questions. Code and data used in the study are available at: https://github.com/mjan2021/Dermoscopic-image-classification.git

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

The datasets generated and used during the study reported in the paper are available through the following github repository: https://github.com/mjan2021/Dermoscopic-image-classification.git

References

  1. Divya G, Liang Q, Wang S, Zhu X (2021) An Empirical Study of Deep Learning Frameworks for Melanoma Cancer Detection using Transfer Learning and Data Augmentation. In 2021 IEEE International Conference on Big Knowledge (ICBK), pp. 38–45. IEEE

  2. Ali K, Shaikh ZA, Khan AA, Laghari AA. Multiclass skin cancer classification using efficientNets–a first step towards preventing skin cancer. Neurosci Inf 2022;2(4):100034

  3. Devansh B, Choromanska A, Berman RS, Stein JA, Polsky D (2019) Towards automated melanoma detection with deep learning: Data purification and augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0

  4. Diallo, Papa Abdou Karim Karou, and Yun Ju. ”Accurate detection of covid-19 using k-efficientnet deep learning image classifier and k-covid chest x-ray images dataset.” In 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp. 1527–1531. IEEE, 2020.

  5. Vasconcelos CN, Nader Vasconcelos B (2017) Convolutional neural network committees for melanoma classification with classical and expert knowledge based image transforms data augmentation.” arXiv preprint arXiv:1702.07025

  6. Salekshahrezaee Z, Leevy JL, Khoshgoftaar TM (2021) Feature extraction for class imbalance using a convolutional autoencoder and data sampling. In 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 217–223. IEEE

  7. Odena A, Olah C, Shlens J. Conditional image synthesis with auxiliary classifier gans. International conference on machine learning. 2017(pp. 2642-2651). PML

  8. Liu Z, Mao H, Wu C-Y, Feichtenhofer C, Darrell T, Xie S (2022) A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986

  9. Ze L, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022

  10. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Doll´ar P, Zitnick CL (2014) ”Microsoft coco: Common objects in context.” In European conference on computer vision, pp. 740–755. Springer, Cham

  11. Atila U, Uçar M, Akyol K, Uçar E (2021) Plant leaf disease classification using EfficientNet deep learning model. Ecol Inform 61:101182

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

  13. Tschandl P, Rosendahl C, Kittler H (2018) The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1):1–9

    Article  Google Scholar 

  14. Jan MT, Hashemi A, Jang J, Yang K, Zhai J, Newman D, Tappen R, Furht B (2023) Non-intrusive Drowsiness Detection Techniques and Their Application in Detecting Early Dementia in Older Drivers. In Proceedings of the Future Technologies Conference, pp. 776–796. Springer, Cham

  15. Alsaidi M, Altaher AS, Tanveer Jan M, Altaher A, Salekshahrezaee Z (2022) COVID-19 Classification Using Deep Learning Two-Stage Approach. arXiv preprint arXiv:2211.15817

  16. Xiang A, Wang F (2019) Towards interpretable skin lesion classification with deep learning models. In AMIA annual symposium proceedings, vol. 2019, p. 1246. American Medical Informatics Association

  17. Chaturvedi SS, Tembhurne JV, Diwan T (2020) A multi-class skin Cancer classification using deep convolutional neural networks. Multimed Tools Appl 79(39-40):28477–28498

    Article  Google Scholar 

  18. Harangi B (2018) Skin lesion classification with ensembles of deep convolutional neural networks. J Biomed Inform 86:25–32

    Article  Google Scholar 

  19. Nyíri T, Kiss A (2018) Novel ensembling methods for dermatological image classification. In International conference on theory and practice of natural computing, pp. 438–448. Springer, Cham

  20. Shahin AH, Kamal A, Elattar MA (2018) Deep ensemble learning for skin lesion classification from dermoscopic images. In 2018 9th Cairo International Biomedical Engineering Conference (CIBEC), pp. 150–153. IEEE

  21. Menegola A, Fornaciali M, Pires R, Bittencourt FV, Avila S, Valle E. Knowledge transfer for melanoma screening with deep learning. In: 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017) 2017 Apr 18 (pp. 297–300). IEEE

  22. Pomponiu V, Nejati H, Cheung N-M (2016) Deepmole: Deep neural networks for skin mole lesion classification. In 2016 IEEE international conference on image processing (ICIP), pp. 2623–2627. IEEE

  23. Milton Md AA (2019) Automated skin lesion classification using ensemble of deep neural networks in ISIC 2018: Skin lesion analysis towards melanoma detection challenge. arXiv preprint arXiv:1901.10802

  24. Hasan HA, Ibrahim AA (2020) Hybrid Detection Techniques for Skin Cancer Images. In 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), pp. 1–8. IEEE

  25. Zhou B, Zhao H, Puig X, Xiao T, Fidler S, Barriuso A, Torralba A (2019) Semantic understanding of scenes through the ade20k dataset. Int J Comput Vision 127(3):302–321

    Article  Google Scholar 

  26. Jha D, Riegler MA, Johansen D, Halvorsen P, Johansen HD (2020) Doubleu-net: A deep convolutional neural network for medical image segmentation. In 2020 IEEE 33rd International symposium on computer-based medical systems (CBMS), pp. 558–564. IEEE

  27. Gessert N, Nielsen M, Shaikh M, Werner R, Schlaefer A (2020) Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. Methods 7:100864

    Google Scholar 

  28. Yao P, Shen S, Mengjuan Xu, Liu P, Zhang F, Xing J, Shao P, Kaffenberger B, Ronald XXu (2021) Single model deep learning on imbalanced small datasets for skin lesion classification. IEEE Trans Med Imaging 41(5):1242–1254

    Article  Google Scholar 

  29. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  30. Goodfellow I, Pouget-Abadie J, Mirza M, Bing Xu, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  31. Murdoch WJ, Singh C, Kumbier K, Abbasi-Asl R, Yu B (2019) Interpretable machine learning: definitions, methods, and applications. arXiv preprint arXiv:1901.04592

  32. Du M, Liu N, Xia Hu (2019) Techniques for interpretable machine learning. Commun ACM 63(1):68–77

    Article  Google Scholar 

  33. Gajera HK, Nayak DR, Zaveri MA (2023) A comprehensive analysis of dermoscopy images for melanoma detection via deep CNN features. Biomed Signal Process Control 79:104186

    Article  Google Scholar 

  34. Alenezi F, Armghan A, Polat K (2023) A multi-stage melanoma recognition framework with deep residual neural network and hyperparameter optimizationbased decision support in dermoscopy images. Expert Syst Appl 215:119352

    Article  Google Scholar 

  35. Emara T, Afify HM, Ismail FH, Hassanien AE (2019) A modified inception-v4 for imbalanced skin cancer classification dataset. In 2019 14th International Conference on Computer Engineering and Systems (ICCES), pp. 28–33. IEEE

  36. Jan MT, Moshfeghi S, Conniff JW, Jang J, Yang K, Zhai J, Rosselli M, Newman D, Tappen R, Furht B (2023) Methods and Tools for Monitoring Driver’s Behavior. arXiv preprint arXiv:2301.12269

  37. Chen K, Zhuang D, Morris Chang J (2022) SuperCon: Supervised contrastive learning for imbalanced skin lesion classification. arXiv preprint arXiv:2202.05685

  38. Ozturk S, Cukur T (2022) Deep clustering via center-oriented margin free-triplet loss for skin lesion detection in highly imbalanced datasets. IEEE J Biomed Health Inform 26(9):4679–4690

    Article  Google Scholar 

  39. Qian S, Ren K, Zhang W, Ning H (2022) Skin lesion classification using CNNs with grouping of multi-scale attention and class-specific loss weighting. Comput Methods Programs Biomed 226:107166

    Article  Google Scholar 

  40. Shen S, Xu M, Zhang F, Shao P, Liu H, Xu L, Zhang C et al (2022) A low-cost high-performance data augmentation for deep learning-based skin lesion classification. BME Frontiers 2022

  41. Baur, Christoph, Shadi Albarqouni, and Nassir Navab. ”MelanoGANs: high resolution skin lesion synthesis with GANs.” arXiv preprint arXiv:1804.04338 (2018).

  42. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  43. Yu Y, Gong Z, Zhong Ping, Shan J (2017) Unsupervised representation learning with deep convolutional neural network for remote sensing images. In Image and Graphics: 9th International Conference, ICIG 2017, Shanghai, China, September 13–15, 2017, Revised Selected Papers, Part II 9, pp. 97–108. Springer International Publishing

  44. Gao H, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708

  45. Qin Z, Liu Z, Zhu P, Xue Y (2020) A GAN-based image synthesis method for skin lesion classification. Comput Methods Programs Biomed 195:105568

    Article  Google Scholar 

  46. Mukti, Zahan I, Biswas D (2019)Transfer learning based plant diseases detection using ResNet50. In 2019 4th International conference on electrical information and communication technology (EICT), pp. 1–6. IEEE

  47. Suhita Ray (2018) Disease classification within dermascopic images using features extracted by resnet50 and classification through deep forest. arXiv preprint arXiv:1807.05711

  48. Shabbir A, Ali N, Ahmed J, Zafar B, Rasheed A, Sajid M, Ahmed A, Hanif Dar S (2021) Satellite and scene image classification based on transfer learning and fine tuning of ResNet50. Mathematical Problems in Engineering 2021:1–18

    Google Scholar 

  49. Council of the European Union (2016) Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), available at http://data.europa.eu/eli/reg/2016/679/2016-05-04, Accessed 21 February 2023

  50. Selvaraju RR, Cogswell M, Das Ak, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618–626

  51. Lucieri, A, Bajwa MN, Braun SA, Malik MI, Dengel A, Ahmed S (2020) On interpretability of deep learning based skin lesion classifiers using concept activation vectors. In 2020 international joint conference on neural networks (IJCNN), pp. 1–10. IEEE

  52. Ulus C, Wang Z, Iqbal SMA, Khan KMdS, Zhu X (2022) Transfer Na¨ıve Bayes Learning using Augmentation and Stacking for SMS Spam Detection. In 2022 IEEE International Conference on Knowledge Graph (ICKG), pp. 275–282. IEEE

  53. Altaher A, Salekshahrezaee Z, Zadeh AA, Rafieipour H, Altaher A (2020) Using multi-inception CNN for face emotion recognition. Journal of Bioengineering Research 3, no. 1: 1–12.

  54. Abidalkareem, AJ, Abd MA, Ibrahim AK, Zhuang H, Altaher AS, Muhamed A (2020.) Diabetic retinopathy (DR) severity level classification using multimodel convolutional neural networks. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 1404–1407. IEEE

  55. Limeros SC, Majchrowska S, Zoubi MK, Ros´en A, Suvilehto J, Sj¨oblom L, Kjellberg M (2022) GAN-based generative modelling for dermatological applications– comparative study. In Arxiv, arXiv:2208.11702

  56. Yosinski J, Clune J, Bengio Y, Lipson H (2014) Howtransferable are features in deep neural networks? In Proc. of NIPS

  57. Man Wu, Wang S, Pan S, Terentis AC, Strasswimmer J, Zhu X (2021) Deep learning data augmentation for Raman spectroscopy cancer tissue classification. Sci Rep 11:23842

    Article  Google Scholar 

  58. Brinker TJ, Hekler A, Utikal JS, Grabe N, Schadendorf D, Klode J, Berking C, Steeb T, Enk AH (2018) Christof von Kalle. Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review, J Med Int Res 20(10):e11936

    Google Scholar 

  59. Sikkandar Y, Mohamed, Alrasheadi BA, Prakash NB, Hemalakshmi GR, Mohanarathinam A, Shankar K (2021) Deep learning based an automated skin lesion segmentation and intelligent classification model. J Ambient Intell humanized Comput 12:3245-3255.

  60. Mirikharaji Z, Abhishek K, Bissoto A, Barata C, Avila S, Valle E, Celebi ME, Hamarneh G (2023) A survey on deep learning for skin lesion segmentation. Med Image Anal: 102863

  61. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440

  62. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and ComputerAssisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18, pp. 234–241. Springer International Publishing

  63. Pollastri F, Bolelli F, Paredes R, Grana C (2020) Augmenting data with GANs to segment melanoma skin lesions. Multimedia Tools and Applications 79:15575–15592

    Article  Google Scholar 

  64. Hasan MdK, Dahal L, Samarakoon PN, Tushar FI, Mart´I R ((2020)) DSNet: Automatic dermoscopic skin lesion segmentation. Computers in biology and medicine 120: 103738

  65. Canalini L, Pollastri F, Bolelli F, Cancilla M, Allegretti S, Grana C (2019) Skin lesion segmentation ensemble with diverse training strategies. In Computer Analysis of Images and Patterns: 18th International Conference, CAIP 2019, Salerno, Italy, September 3–5, 2019, Proceedings, Part I 18, pp. 89–101. Springer International Publishing

  66. Seifallahi M, HasaniMehraban A, Galvin JE, Ghoraani B (2022) Alzheimer’s disease detection using comprehensive analysis of Timed Up and Go test via Kinect V. 2 camera and machine learning. IEEE Trans Neural Syst Rehabil Eng 30:1589–1600

    Article  Google Scholar 

  67. Soudani A, Barhoumi W (2019) An image-based segmentation recommender using crowdsourcing and transfer learning for skin lesion extraction. Expert Syst Appl 118:400–410

    Article  Google Scholar 

  68. Xie Y, Zhang J, Xia Y, Shen C (2020) A mutual bootstrapping model for automated skin lesion segmentation and classification. IEEE Trans Med Imaging 39(7):2482–2493

    Article  Google Scholar 

  69. Jin Q, Cui H, Sun C, Meng Z, Ran Su (2021) Cascade knowledge diffusion network for skin lesion diagnosis and segmentation. Appl Soft Comput 99:106881

    Article  Google Scholar 

  70. Lei B, Xia Z, Jiang F, Jiang X, Ge Z, Yanwu Xu, Qin J, Chen S, Wang T, Wang S (2020) Skin lesion segmentation via generative adversarial networks with dual discriminators. Med Image Anal 64:101716

    Article  Google Scholar 

  71. Tu W, Liu X, Wei Hu, Pan Z (2019) Dense-residual network with adversarial learning for skin lesion segmentation. IEEE Access 7:77037–77051

    Article  Google Scholar 

  72. Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille L, Zhou Y (2021) Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306

  73. Gulzar Y, Khan SA (2022) Skin lesion segmentation based on vision transformers and convolutional neural networks—A comparative study. Appl Sci 12(12):5990

    Article  Google Scholar 

  74. Iqbal SMA, Mahgoub I, Du E, Leavitt MA, Asghar W (2022) Development of a wearable belt with integrated sensors for measuring multiple physiological parameters related to heart failure. Sci Rep 12(1):20264

    Article  Google Scholar 

  75. Iqbal SMA, Asghar W (2023) Smartphone Integration with Point-of-Care Devices for Disease Diagnostics. In Emerging Technologies In Biophysical Sciences: A World Scientific Reference: Volume 3: Emerging Technologies for Diagnostics, pp. 317–335

  76. Jasil SPG, Ulagamuthalvi V (2021) Deep learning architecture using transfer learning for classification of skin lesions. J Ambient Intell Humanized Comput: 1–8

  77. Khan Attique, Muhammad Muhammad Sharif, Akram Tallha, Kadry Seifedine, Hsu Ching-Hsien (2022) A two-stream deep neural network-based intelligent system for complex skin cancer types classification. Int J Intell Syst 37(12):10621–10649

    Article  Google Scholar 

  78. Naqvi Maryam, Gilani Syed Qasim, Syed Tehreem, Marques Oge, Kim Hee-Cheol (2023) Skin Cancer Detection Using Deep Learning-A Review. Diagnostics (Basel, Switzerland) 13(11):1911

    Google Scholar 

  79. Qasim Gilani S, Syed T, Umair M, Marques O. Skin Cancer Classification Using Deep Spiking Neural Network. J Digit Imaging (2023): 1–11

  80. Gilani SQ, Marques O (2023) Skin lesion analysis using generative adversarial networks: A review. Multimedia Tools Appl: 1–42

Download references

Acknowledgements

This research is partially sponsored by the U.S. National Science Foundation under grant No. IIS-2302786. Special thanks to Zahra Salekshahrezae, Rahmi Alagoz, Ali Salem Altaher, and Nathan Guan for their contributions to the reformatting and proofreading of the manuscript.

Author information

Author notes

  1. Mostapha Alsaidi and Muhammad Tanveer Jan are equal contributors.

Authors and Affiliations

  1. Department. of Electrical Engineering & Computer Science, Florida Atlantic University, Boca Raton, FL, 33431, USA

    Mostapha Alsaidi, Muhammad Tanveer Jan, Ahmed Altaher, Hanqi Zhuang & Xingquan Zhu

Authors
  1. Mostapha Alsaidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Tanveer Jan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Altaher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hanqi Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xingquan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Tanveer Jan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alsaidi, M., Jan, M.T., Altaher, A. et al. Tackling the class imbalanced dermoscopic image classification using data augmentation and GAN. Multimed Tools Appl 83, 49121–49147 (2024). https://doi.org/10.1007/s11042-023-17067-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17067-1

Keywords

Search

Navigation