Skip to main content
Book cover

International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging

International Workshop on Preterm, Perinatal and Paediatric Image Analysis

UNSURE 2021, PIPPI 2021: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis pp 112–121Cite as

Robust Selective Classification of Skin Lesions with Asymmetric Costs

  • 912 Accesses

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12959)

Abstract

Automated image analysis of skin lesions has potential to improve diagnostic decision making. A clinically useful system should be selective, rejecting images it is ill-equipped to classify, for example because they are of lesion types not represented well in training data. Furthermore, lesion classifiers should support cost-sensitive decision making. We investigate methods for selective, cost-sensitive classification of lesions as benign or malignant using test images of lesion types represented and not represented in training data. We propose EC-SelectiveNet, a modification to SelectiveNet that discards the selection head at test time, making decisions based on expected costs instead. Experiments show that training for full coverage is beneficial even when operating at lower coverage, and that EC-SelectiveNet outperforms standard cross-entropy training, whether or not temperature scaling or Monte Carlo dropout averaging are used, in both symmetric and asymmetric cost settings.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-87735-4_11
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-87735-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Notes

  1. 1.

    GitHub Repository: https://github.com/UoD-CVIP/Selective_Dermatology.

References

  1. Brinker, T.J., et al.: Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur. J. Cancer 113, 47–54 (2019)

    CrossRef  Google Scholar 

  2. Carse, J., McKenna, S.: Active learning for patch-based digital pathology using convolutional neural networks to reduce annotation costs. In: Reyes-Aldasoro, C.C., Janowczyk, A., Veta, M., Bankhead, P., Sirinukunwattana, K. (eds.) ECDP 2019. LNCS, vol. 11435, pp. 20–27. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23937-4_3

    CrossRef  Google Scholar 

  3. Codella, N., et al.: Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In: IEEE ISBI, pp. 168–172 (2018)

    Google Scholar 

  4. Combalia, M., et al: BCN20000: dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

  5. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)

    Google Scholar 

  6. DeVries, T., Taylor, G.W.: Leveraging uncertainty estimates for predicting segmentation quality. In: Conference on Medical Imaging with Deep Learning (MIDL) (2018)

    Google Scholar 

  7. Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–8 (2017)

    CrossRef  Google Scholar 

  8. Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: Proceedings of the 33rd International Conference on Machine Learning (ICML), vol. PMLR 48, pp. 1050–1059 (2016)

    Google Scholar 

  9. Geifman, Y., El-Yaniv, R.: SelectiveNet: a deep neural network with an integrated reject option. In: Proceedings of the 36th International Conference on Machine Learning (ICML), vol. PMLR 97, pp. 2151–2159 (2019)

    Google Scholar 

  10. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), vol. PMLR 70, pp. 1321–1330 (2017)

    Google Scholar 

  11. Haenssle, H.A., Fink, C., Schneiderbauer, R., et al.: Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29(8), 1836–1842 (2018)

    CrossRef  Google Scholar 

  12. Han, S.S., Kim, M.S., Lim, W., Park, G.H., Park, I., Chang, S.E.: Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm. J. Inv. Dermatol. 138(7), 1529–1538 (2018)

    CrossRef  Google Scholar 

  13. Han, S.S., et al.: Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders. J. Inv. Dermatol. 140(9), 1753–1761 (2020)

    CrossRef  Google Scholar 

  14. Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: ICLR (2017)

    Google Scholar 

  15. Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580 (2012)

  16. Kawahara, J., Hamarneh, G.: Visual diagnosis of dermatological disorders: human and machine performance. arxiv:1906.01256, 6 (2019)

  17. Mårtensson, G., et al.: The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study. Med. Image Anal. 66, 101714 (2020)

    CrossRef  Google Scholar 

  18. Mobiny, A., Singh, A., Van Nguyen, H.: Risk-aware machine learning classifier for skin lesion diagnosis. J. Clin. Med. 8(8), 1241 (2019)

    CrossRef  Google Scholar 

  19. Mozafari, A.S., Gomes, H.S., Leão, W., Janny, S., Gagné, C.: Attended temperature scaling: a practical approach for calibrating deep neural networks. arXiv preprint arXiv:1810.11586 (2018)

  20. Nixon, J., Dusenberry, M.W., Zhang, L., Jerfel, G., Tran, D.: Measuring calibration in deep learning. In: CVPR Workshops, vol. 2 (2019)

    Google Scholar 

  21. Smith, L.: Cyclical learning rates for training neural networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017)

    Google Scholar 

  22. Tan, M., Le, Q.V.: EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning (ICML), vol. PMLR 97, pp. 6105–6114 (2019)

    Google Scholar 

  23. Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5, 180161 (2018)

    CrossRef  Google Scholar 

Download references

Acknowledgments

This paper reports independent research funded by the National Institute for Health Research (Artificial Intelligence, Deep learning for effective triaging of skin disease in the NHS, AI_AWARD01901) and NHSX. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health Research, NHSX or the Department of Health and Social Care. This research was also funded by the Detect Cancer Early programme, and the Discovery Institute of Dermatology.

Author information

Authors and Affiliations

  1. CVIP, School of Science and Engineering, University of Dundee, Scotland, UK

    Jacob Carse, Tamás Süveges, Stephen Hogg, Emanuele Trucco & Stephen McKenna

  2. School of Medicine, Ninewells Hospital and Medical School, Dundee, UK

    Charlotte Proby

  3. Department of Dermatology, Ninewells Hospital and Medical School, Dundee, UK

    Charlotte Proby & Colin Fleming

Authors
  1. Jacob Carse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamás Süveges
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephen Hogg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emanuele Trucco
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Charlotte Proby
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Colin Fleming
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Stephen McKenna
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen McKenna .

Editor information

Editors and Affiliations

  1. University College London/King's College London, London, UK

    Carole H. Sudre

  2. Medical University of Vienna and TU Wien, Vienna, Austria

    Roxane Licandro

  3. University of Tübingen, Tübingen, Germany

    Christian Baumgartner

  4. King's College London, London, UK

    Prof. Dr. Andrew Melbourne

  5. Massachusetts General Hospital, Harvard Medical School, MIT, Cambridge, MA, USA

    Dr. Adrian Dalca

  6. King's College London, London, UK

    Dr. Jana Hutter

  7. Microsoft Research/University College London, London, UK

    Ryutaro Tanno

  8. Boston Children's Hospital, Boston, MA, USA

    Esra Abaci Turk

  9. Technical University Denmark, Kongens Lyngby, Denmark

    Koen Van Leemput

  10. Hewlett Packard, Barcelona, Spain

    Jordina Torrents Barrena

  11. Harvard Medical School/Brigham and Women's Hospital, Boston, MA, USA

    William M. Wells

  12. The Hospital For Sick Children, University of Toronto, Toronto, ON, Canada

    Christopher Macgowan

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 184 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Carse, J. et al. (2021). Robust Selective Classification of Skin Lesions with Asymmetric Costs. In: , et al. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis. UNSURE PIPPI 2021 2021. Lecture Notes in Computer Science(), vol 12959. Springer, Cham. https://doi.org/10.1007/978-3-030-87735-4_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87735-4_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87734-7

  • Online ISBN: 978-3-030-87735-4

  • eBook Packages: Computer ScienceComputer Science (R0)