Skip to main content
SpringerLink
Log in

CIRCLe: Color Invariant Representation Learning for Unbiased Classification of Skin Lesions

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13804))

Included in the following conference series:

Abstract

While deep learning based approaches have demonstrated expert-level performance in dermatological diagnosis tasks, they have also been shown to exhibit biases toward certain demographic attributes, particularly skin types (e.g., light versus dark), a fairness concern that must be addressed. We propose CIRCLe, a skin color invariant deep representation learning method for improving fairness in skin lesion classification. CIRCLe is trained to classify images by utilizing a regularization loss that encourages images with the same diagnosis but different skin types to have similar latent representations. Through extensive evaluation and ablation studies, we demonstrate CIRCLe’s superior performance over the state-of-the-art when evaluated on 16k+ images spanning 6 Fitzpatrick skin types and 114 diseases, using classification accuracy, equal opportunity difference (for light versus dark groups), and normalized accuracy range, a new measure we propose to assess fairness on multiple skin type groups. Our code is available at https://github.com/arezou-pakzad/CIRCLe.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adamson, A.S., Smith, A.: Machine learning and health care disparities in dermatology. JAMA Dermatol. 154(11), 1247–1248 (2018)

    Article  Google Scholar 

  2. Adamson, A.S., Suarez, E.A., Welch, H.G.: Estimating overdiagnosis of melanoma using trends among black and white patients in the US. JAMA Dermatol. 158(4), 426–431 (2022)

    Article  Google Scholar 

  3. Adelekun, A., Onyekaba, G., Lipoff, J.B.: Skin color in dermatology textbooks: an updated evaluation and analysis. J. Am. Acad. Dermatol. 84(1), 194–196 (2021)

    Article  Google Scholar 

  4. Agbai, O.N., et al.: Skin cancer and photoprotection in people of color: a review and recommendations for physicians and the public. J. Am. Acad. Dermatol. 70(4), 748–762 (2014)

    Article  Google Scholar 

  5. Angwin, J., Larson, J., Mattu, S., Kirchner, L.: Machine bias. In: Ethics of Data and Analytics, pp. 254–264 (2016)

    Google Scholar 

  6. Ballerini, L., Fisher, R.B., Aldridge, B., Rees, J.: A color and texture based hierarchical K-NN approach to the classification of non-melanoma skin lesions. In: Color Medical Image Analysis, pp. 63–86 (2013)

    Google Scholar 

  7. Barata, C., Marques, J.S., Emre Celebi, M.: Deep attention model for the hierarchical diagnosis of skin lesions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 2757–2765 (2019)

    Google Scholar 

  8. Bellamy, R.K., et al.: AI fairness 360: an extensible toolkit for detecting and mitigating algorithmic bias. IBM J. Res. Dev. 63(4/5), 1–4 (2019)

    Article  Google Scholar 

  9. Bevan, P.J., Atapour-Abarghouei, A.: Detecting melanoma fairly: skin tone detection and debiasing for skin lesion classification. arXiv preprint arXiv:2202.02832 (2022)

  10. Brinker, T.J., et al.: A convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task. Eur. J. Cancer 111, 148–154 (2019)

    Article  Google Scholar 

  11. Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency, pp. 77–91 (2018)

    Google Scholar 

  12. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2018)

    Google Scholar 

  13. Codella, N., et al.: Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC). arXiv preprint arXiv:1902.03368 (2019)

  14. Codella, N.C., et al.: Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC). In: 2018 IEEE 15th International Symposium on Biomedical Imaging, pp. 168–172 (2018)

    Google Scholar 

  15. Combalia, M., et al.: BCN20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

  16. Daneshjou, R., Smith, M.P., Sun, M.D., Rotemberg, V., Zou, J.: Lack of transparency and potential bias in artificial intelligence data sets and algorithms: a scoping review. JAMA Dermatol. 157(11), 1362–1369 (2021)

    Article  Google Scholar 

  17. Daneshjou, R., et al.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci. Adv.8(31), eabq6147 (2022)

    Google Scholar 

  18. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  19. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012)

    Google Scholar 

  20. Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)

    Article  Google Scholar 

  21. Fitzpatrick, T.B.: The validity and practicality of sun-reactive skin types I through VI. Arch. Dermatol. 124(6), 869–871 (1988)

    Article  Google Scholar 

  22. Giotis, I., Molders, N., Land, S., Biehl, M., Jonkman, M.F., Petkov, N.: MED-NODE: a computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert Syst. Appl. 42(19), 6578–6585 (2015)

    Article  Google Scholar 

  23. Groh, M., et al.: Evaluating deep neural networks trained on clinical images in dermatology with the Fitzpatrick 17k dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1820–1828 (2021)

    Google Scholar 

  24. Haenssle, H., et al.: Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29(8), 1836–1842 (2018)

    Article  Google Scholar 

  25. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. Adv. Neural. Inf. Process. Syst. 29, 3323–3331 (2016)

    Google Scholar 

  26. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  27. Howard, A., et al.: Searching for MobileNetV3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  28. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  29. Kawahara, J., Daneshvar, S., Argenziano, G., Hamarneh, G.: Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE J. Biomed. Health Inform. 23(2), 538–546 (2019)

    Article  Google Scholar 

  30. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  31. Kinyanjui, N.M., et al.: Fairness of classifiers across skin tones in dermatology. In: Medical Image Computing and Computer-Assisted Intervention, pp. 320–329 (2020)

    Google Scholar 

  32. Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., Mullainathan, S.: Human decisions and machine predictions. Q. J. Econ. 133(1), 237–293 (2018)

    MATH  Google Scholar 

  33. Kleinberg, J., Ludwig, J., Mullainathan, S., Rambachan, A.: Algorithmic fairness. In: American Economic Association Papers and Proceedings, vol. 108, pp. 22–27 (2018)

    Google Scholar 

  34. Lafarge, M.W., Pluim, J.P., Eppenhof, K.A., Veta, M.: Learning domain-invariant representations of histological images. Front. Med. 6, 162 (2019)

    Article  Google Scholar 

  35. Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl. Acad. Sci. 117(23), 12592–12594 (2020)

    Article  Google Scholar 

  36. Lester, J., Jia, J., Zhang, L., Okoye, G., Linos, E.: Absence of images of skin of colour in publications of COVID-19 skin manifestations. Br. J. Dermatol. 183(3), 593–595 (2020)

    Article  Google Scholar 

  37. Liu, Q., Chen, C., Dou, Q., Heng, P.A.: Single-domain generalization in medical image segmentation via test-time adaptation from shape dictionary. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, pp. 1756–1764 (2022)

    Google Scholar 

  38. Liu, Y., et al.: A deep learning system for differential diagnosis of skin diseases. Nat. Med. 26(6), 900–908 (2020)

    Article  Google Scholar 

  39. Mendonça, T., Ferreira, P.M., Marques, J.S., Marcal, A.R., Rozeira, J.: PH\(^2\)–a dermoscopic image database for research and benchmarking. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5437–5440 (2013)

    Google Scholar 

  40. Muandet, K., Balduzzi, D., Schölkopf, B.: Domain generalization via invariant feature representation. In: International Conference on International Conference on Machine Learning, pp. 10–18 (2013)

    Google Scholar 

  41. Nguyen, A.T., Tran, T., Gal, Y., Baydin, A.G.: Domain invariant representation learning with domain density transformations. Adv. Neural. Inf. Process. Syst. 34, 5264–5275 (2021)

    Google Scholar 

  42. Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S.: Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464), 447–453 (2019)

    Article  Google Scholar 

  43. Osto, M., Hamzavi, I.H., Lim, H.W., Kohli, I.: Individual typology angle and Fitzpatrick skin phototypes are not equivalent in photodermatology. Photochem. Photobiol. 98(1), 127–129 (2022)

    Article  Google Scholar 

  44. Pacheco, A.G., et al.: PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones. Data Brief 32, 106221 (2020)

    Article  Google Scholar 

  45. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035 (2019)

    Google Scholar 

  46. Prince, S.: Tutorial #1: Bias and fairness in AI (2019). https://www.borealisai.com/en/blog/tutorial1-bias-and-fairness-ai/. Accessed 14 Apr 2022

  47. Puyol-Antón, E., et al.: Fairness in cardiac MR image analysis: an investigation of bias due to data imbalance in deep learning based segmentation. In: Medical Image Computing and Computer Assisted Intervention, pp. 413–423 (2021)

    Google Scholar 

  48. Rotemberg, V., et al.: A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci. Data 8(1), 1–8 (2021)

    Google Scholar 

  49. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  50. Seyyed-Kalantari, L., Liu, G., McDermott, M., Chen, I.Y., Ghassemi, M.: CheXclusion: fairness gaps in deep chest X-ray classifiers. In: Biocomputing 2021: Proceedings of the Pacific Symposium, pp. 232–243 (2020)

    Google Scholar 

  51. Seyyed-Kalantari, L., Zhang, H., McDermott, M., Chen, I.Y., Ghassemi, M.: Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat. Med. 27(12), 2176–2182 (2021)

    Article  Google Scholar 

  52. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  53. Sun, X., Yang, J., Sun, M., Wang, K.: A benchmark for automatic visual classification of clinical skin disease images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 206–222. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_13

    Chapter  Google Scholar 

  54. Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5(1), 1–9 (2018)

    Article  Google Scholar 

  55. Ware, O.R., Dawson, J.E., Shinohara, M.M., Taylor, S.C.: Racial limitations of Fitzpatrick skin type. Cutis 105(2), 77–80 (2020)

    Google Scholar 

  56. Weiss, E.B.: Brown skin matters. https://brownskinmatters.com/. Accessed 23 Jun 2022

  57. Wu, Y., Zeng, D., Xu, X., Shi, Y., Hu, J.: FairPrune: achieving fairness through pruning for dermatological disease diagnosis. arXiv preprint arXiv:2203.02110 (2022)

  58. Yang, J., et al.: Self-paced balance learning for clinical skin disease recognition. IEEE Trans. Neural Networks Learn. Syst. 31(8), 2832–2846 (2019)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

We would like to thank lab members Jeremy Kawahara and Ashish Sinha for their helpful discussions and comments on this work. We would also like to thank the reviewers for their valuable feedback that helped in improving this work. This project was partially funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), and its computational resources were provided by NVIDIA and Compute Canada (computecanada.ca).

Author information

Authors and Affiliations

  1. School of Computing Science, Simon Fraser University, Burnaby, Canada

    Arezou Pakzad, Kumar Abhishek & Ghassan Hamarneh

Authors
  1. Arezou Pakzad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kumar Abhishek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ghassan Hamarneh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arezou Pakzad .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pakzad, A., Abhishek, K., Hamarneh, G. (2023). CIRCLe: Color Invariant Representation Learning for Unbiased Classification of Skin Lesions. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13804. Springer, Cham. https://doi.org/10.1007/978-3-031-25069-9_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25069-9_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25068-2

  • Online ISBN: 978-3-031-25069-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation