Skip to main content
SpringerLink
Log in

AMTLUS: Attention-guided multi-task learning with uncertainty estimation in skin lesion segmentation and classification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Skin lesion segmentation and classification from dermoscopic images have emerged as pivotal research topics, playing vibrant role in early detection and diagnosis of skin diseases, including melanoma. Previous studies have employed various deep learning models for skin lesion segmentation and classification, enabling the automatic learning of complex and discriminative features from dermoscopic images. However, inherent challenges arise due to the variance in skin lesion shape, size, and contrast, leading to intrinsic limitations of former models, such as Isolated Representation Learning, Uniform Attention, Limited Model Generalization, Reduced Model Interpretability, and Uncertainty. To address these limitations and propel the field forward, this paper introduces a novel frameworkcalled AMTLUS that leverages Multi-Task Learning (MTL) in conjunction with deep Attention Mechanisms and Uncertainty Estimation. The integration of MTL facilitates joint training of segmentation and classification tasks, enabling shared representation learning and efficient utilization of data. Incorporating attention mechanisms dynamically focuses on informative regions within dermoscopic images, improving segmentation accuracy and feature extraction for classification. Uncertainty estimation techniques quantify model confidence, offering probabilistic interpretations for improved reliability and interpretability. Our widespread experiments conducted on the ISIC-2016 dataset demonstrate superior accuracy and reliability, showcasing the proposed model’s capability to identify challenging cases. This deep learning framework represents a significant advancement in automated skin lesion analysis, enhancing early detection and diagnosis of skin diseases, including melanoma.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data Availability

Data will be made available on request.

References

  1. Habif TP, Campbell JL, Dinulos JG, Chapman MS, Zug KA. (2017) Skin Disease: Diagnosis and Treatment, 4th Edn. Elsevier Health Sciences,eBook . Available at: https://shop.elsevier.com/books/skin-disease/habif/978-0-323-44222-0

  2. Hasan MK, Ahamad MA, Yap CH, Yang G (2023) A survey, review, and future trends of skin lesion segmentation and classification. Comput Biol Med 155:106624. https://doi.org/10.1016/j.compbiomed.2023.106624

  3. American Cancer Society Editorial Team (2023) Key Statistics for Melanoma Skin Cancer. Facts & Figures 2023. American Cancer Society, cancer.org - 1.800.227.2345. Atlanta, Ga. Available at: https://www.cancer.org/content/dam/CRC/PDF/Public/8823.00.pdf

  4. Emre Celebi M, Codella Noel, Halpern Allan (2019) Dermoscopy image analysis: overview and future directions. IEEE Journal Of Biomedical And Health Informatics 2(2):474–478

    Google Scholar 

  5. Adegun A, Viriri S (2021) Deep learning techniques for skin lesion analysis and melanoma cancer detection: a survey of state-of-the-art. Artif Intell Rev 54:811–841

    Article  Google Scholar 

  6. Milton MA (2019) Automated skin lesion classification using ensemble of deep neural networks in ISIC 2018: Skin lesion analysis towards melanoma detection challenge.  Eprint-1901.10802. https://doi.org/10.48550/arXiv.1901.10802

  7. Bansal P, Garg R, Soni P (2022) Detection of melanoma in dermoscopic images by integrating features extracted using handcrafted and deep learning models. Comput Ind Eng 168:108060

    Article  Google Scholar 

  8. Wu H, Chen S, Chen G, Wang W, Lei B, Wen Z (2022) FAT-Net: Feature adaptive transformers for automated skin lesion segmentation. Med Image Anal 76:102327

    Article  Google Scholar 

  9. Löwe S, O'Connor P, Veeling B (2019) Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in neural information processing systems 32. eprint:1905.11786, arXiv. https://doi.org/10.48550/arXiv.1905.11786

  10. Peng J, Wang Ye (2021) Medical image segmentation with limited supervision: a review of deep network models. IEEE Access 9:36827–36851

    Article  Google Scholar 

  11. Jaworek-Korjakowska Joanna, Brodzicki Andrzej, Cassidy Bill, Kendrick Connah, Yap MoiHoon (2021) Interpretability of a deep learning based approach for the classification of skin lesions into main anatomic body sites. Cancers 13(23):6048

    Article  Google Scholar 

  12. Chen S, Yang X, Chen Y, Yu H, Cai H (2022) Uncertainty-based Fusion Netwok for Automatic Skin Lesion Diagnosis.  In: 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, pp 1487–1492. https://doi.org/10.1109/BIBM55620.2022.9994932

  13. Zhang G, Zhao K, Hong Y, Qiu X, Zhang K, Wei B (2021) SHA-MTL: soft and hard attention multi-task learning for automated breast cancer ultrasound image segmentation and classification. Int J Comput Assist Radiol Surg 16:1719–1725

    Article  Google Scholar 

  14. Zhang Y, Li H, Jie Du, Qin J, Wang T, Chen Y, Liu B, Gao W, Ma G, Lei B (2021) 3D multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification. IEEE Trans Med Imaging 40(6):1618–1631

    Article  Google Scholar 

  15. Jungo Alain, and Mauricio Reyes (2019) "Assessing reliability and challenges of uncertainty estimations for medical image segmentation." In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019 Proceedings, Part II 22, pp. 48–56. Springer.

  16. Cheng J, Liu J, Kuang H, Wang J (2022) A fully automated multimodal MRI-based multi-task learning for glioma segmentation and IDH genotyping. IEEE Trans Med Imaging 41(6):1520–1532

    Article  Google Scholar 

  17. Ronneberger O, Fischer P, Brox T (2015) “U-Net: Convolutional Networks for Biomedical Image Segmentation” Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, LNCS 9351:234–241

    Google Scholar 

  18. Esteva A, Kuprel B, Novoa R et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118. https://doi.org/10.1038/nature21056

    Article  Google Scholar 

  19. Wang Y, Feng Y, Zhang L, Zhou JT, Liu Y, Goh RS, Zhen L (2022) Adversarial multimodal fusion with attention mechanism for skin lesion classification using clinical and dermoscopic images. Med Image Anal 81:102535. https://doi.org/10.1016/j.media.2022.102535

  20. Parvathi SSL, Chandana BS, Harikiran J (2023). Depth invariant 3D-CU-Net model with completely connected dense skip networks for MRI kidney tumor segmentation. Traitement du Signal, 40,(1)217–225. https://doi.org/10.18280/ts.400120.

  21. Singh RK, Gorantla R, Allada SGR, Narra P (2022) SkiNet: A deep learning framework for skin lesion diagnosis with uncertainty estimation and explainability. PLoS ONE 17(10):e0276836. https://doi.org/10.1371/journal.pone.0276836

    Article  Google Scholar 

  22. Bissoto A, Valle E, Avila S (2021) Gan-based data augmentation and anonymization for skin-lesion analysis: A critical review. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1847–1856

  23. Carvalho R, Pedrosa J, Nedelcu T (2021) Multimodal multi-tasking for skin lesion classification using deep neural networks. In: Bebis G et al (eds) Advances in Visual Computing. ISVC 2021. Lecture Notes in Computer Science, vol 13017. Springer, https://doi.org/10.1007/978-3-030-90439-5_3

  24. Sun L, Zhang Y, Liu T, Ge H, Tian J, Qi X, Sun J, Zhao Y (2023) A collaborative multi-task learning method for BI-RADS category 4 breast lesion segmentation and classification of MRI images. Comput Methods Programs Biomed 240:107705

    Article  Google Scholar 

  25. He X, Wang Y, Zhao S, Chen X (2023) Joint segmentation and classification of skin lesions via a multi-task learning convolutional neural network. System Expert Appl 230:120174. https://doi.org/10.1016/j.eswa.2023.120174

  26. Xiaoyuan Yu, Tang Suigu, Cheang Chak Fong, Hon Ho Yu, Cheong Choi I (2021) Multi-task model for esophageal lesion analysis using endoscopic images: classification with image retrieval and segmentation with attention. Sensors 22(1):283

    Article  Google Scholar 

  27. Lyu J, Sui B, Wang C, Dou Q, Qin J (2023) Adaptive feature aggregation based multi-task learning for uncertainty-guided semi-supervised medical image segmentation. System Expert Appl 232:120836. https://doi.org/10.1016/j.eswa.2023.120836

  28. Ren K et al (2023) Uncertainty-informed mutual learning for joint medical image classification and segmentation. In: Greenspan H et al (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14223. Springer, Cham. https://doi.org/10.1007/978-3-031-43901-8_4

  29. Hu L, Li J, Peng X, Xiao J, Zhan Bo, Chen Zu, Xi Wu, Zhou J, Wang Y (2022) Semi-supervised NPC segmentation with uncertainty and attention guided consistency. Knowl-Based Syst 239:108021

    Article  Google Scholar 

  30. Gutman David, Codella Noel CF, Celebi Emre, Helba Brian, Marchetti Michael, Mishra Nabin, Halpern Allan (2016) "Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)". eprint arXiv:1605.01397.

  31. M Tan, QV Le, EfficientNet: Rethinking model scaling for convolutional neural networks, 2019, https://doi.org/10.48550/ARXIV.1905.11946.

  32. Wang Y, Su J, Xu Q, Zhong Y (2023) A Collaborative Learning Model for Skin Lesion Segmentation and Classification. Diagnostics 13:912. https://doi.org/10.3390/diagnostics13050912

    Article  Google Scholar 

  33. Hu H, Li Q, Zhao Y, Zhang Ye (2020) Parallel deep learning algorithms with hybrid attention mechanism for image segmentation of lung tumors. IEEE Trans Industr Inf 17(4):2880–2889

    Article  Google Scholar 

  34. Dolezal JM, Srisuwananukorn A, Karpeyev D et al (2022) Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology. Nat Commun 13:6572. https://doi.org/10.1038/s41467-022-34025-x

    Article  Google Scholar 

  35. Maharana K, Mondal S, Nemade B (2022) A review: Data pre-processing and data augmentation techniques. Global Trans Proc 3(1):91–99

    Article  Google Scholar 

  36. Dai Xianjin, Lei Yang, Liu Yingzi et al (2020) Intensity non-uniformity correction in MR imaging using residual cycle generative adversarial network. Phys Med Biol 65(21):215025

    Article  Google Scholar 

  37. Gessert N, Nielsen M, Shaikh M, Werner R, Schlaefer A (2020) Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. MethodsX 7:100864

    Article  Google Scholar 

  38. Alvar SR, Bajić IV (2019) Multi-task learning with compressible features for collaborative intelligence. In: 2019 IEEE International Conference on Image Processing (ICIP), IEEE, pp 1705–1709. https://ieeexplore.ieee.org/document/8803110

  39. Qin Xu, Wang Z, Bai Y, Xie X, Jia H (2020) FFA-Net: Feature fusion attention network for single image dehazing. Proc AAAI Conf Artif Intell 34(07):11908–11915

    Google Scholar 

  40. Chefer H, Gur S, Wolf L (2021) Transformer interpretability beyond attention visualization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 782–791.  https://ieeexplore.ieee.org/document/9577970

  41. Yao J, Pan W, Ghosh S, Doshi-Velez F (2019) Quality of uncertainty quantification for Bayesian neural network inference. In: Proceedings at the International Conference on Machine Learning: Workshop on Uncertainty & Robustness in Deep Learning (ICML). https://api.semanticscholar.org/CorpusID:195345113

  42. Lahlou S, Jain M, Nekoei H, Butoi VI, Bertin P, Rector-Brooks J, Korablyov M, Bengio Y. (2021) Deup: Direct epistemic uncertainty prediction. ArXiv abs/2102.08501.  https://doi.org/10.48550/arXiv.2102.08501

  43. van de Schoot Rens, Depaoli Sarah, King Ruth, Kramer Bianca, Märtens Kaspar, Tadesse Mahlet G, Vannucci Marina et al (2021) Bayesian statistics and modelling. Nat Rev Methods Prim 1(1):1

    Article  Google Scholar 

  44. H Awano, M Hashimoto (2020) BYNQNet: Bayesian neural network with quadratic activations for sampling-free uncertainty estimation on FPGA. In: 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp 1402–1407. https://ieeexplore.ieee.org/document/9116302

  45. Muller D, Soto-Rey I, Kramer F (2022) Towards a guideline for evaluation metrics in medical image segmentation. BMC Res Notes 15(1):1–8

    Article  Google Scholar 

  46. Ghorbel I, Rossant F, Bloch I, Tick S, Paques M (2011) Automated segmentation of macular layers in OCT images and quantitative evaluation of performances. Pattern Recogn 44(8):1590–1603

    Article  Google Scholar 

  47. Yang Li, Shami A (2020) On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing 415:295–316

    Article  Google Scholar 

  48. Nguyen MN, Bui PHD, Nguyen KQ, Nguyen HT (2023). Efficiency Evaluation of Hanning Window-based Filter on Human Skin Disease Diagnosis. In: Nguyen TDL, Verdú E, Le AN, Ganzha M (eds) Intelligent Systems and Networks. ICISN 2023. Lecture Notes in Networks and Systems, vol 752. Springer, Singapore. https://doi.org/10.1007/978-981-99-4725-6_58

  49. Liu Y, Xiong Z, Yuan Y, Wang Q (2023) Distilling Knowledge From Super-Resolution for Efficient Remote Sensing Salient Object Detection. IEEE Trans Geosci Remote Sens 61:1–16. https://doi.org/10.1109/TGRS.2023.3267271

    Article  Google Scholar 

  50. Liu Y, Li Q, Yuan Y, Du Q, Wang Q (2022) ABNet: Adaptive Balanced Network for Multiscale Object Detection in Remote Sensing Imagery. IEEE Trans Geosci Remote Sens 60:1–14. https://doi.org/10.1109/TGRS.2021.3133956

    Article  Google Scholar 

  51. Liu Y, Xiong Z, Yuan Y, Wang Q (2023) Transcending Pixels: Boosting Saliency Detection via Scene Understanding From Aerial Imagery. IEEE Trans Geosci Remote Sens 61:1–16. https://doi.org/10.1109/TGRS.2023.3298661

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, JNTU-GV, Vizianagaram, Andhra Pradesh, India

    Aravinda Kasukurthi & Rajya Lakshmi Davuluri

  2. Department of Computer Science and Business System, R.V.R. & J.C. College of Engineering, Guntur, Andhra Pradesh, India

    Aravinda Kasukurthi

Authors
  1. Aravinda Kasukurthi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajya Lakshmi Davuluri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aravinda Kasukurthi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kasukurthi, A., Davuluri, R. AMTLUS: Attention-guided multi-task learning with uncertainty estimation in skin lesion segmentation and classification. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19360-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19360-z

Keywords

Search

Navigation