Skip to main content
SpringerLink
Log in

Crowd Disagreement About Medical Images Is Informative

  • Conference paper
  • First Online:
Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis (LABELS 2018, CVII 2018, STENT 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11043))

Abstract

Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at https://figshare.com/s/5cbbce14647b66286544.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hussein, S., Cao, K., Song, Q., Bagci, U.: Risk stratification of lung nodules using 3D CNN-based multi-task learning. arXiv preprint arXiv:1704.08797 (2017)

  2. O’Neil, A.Q., Murchison, J.T., van Beek, E.J.R., Goatman, K.A.: Crowdsourcing labels for pathological patterns in CT lung scans: can non-experts contribute expert-quality ground truth? In: Cardoso, M.J., et al. (eds.) LABELS/CVII/STENT -2017. LNCS, vol. 10552, pp. 96–105. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67534-3_11

    Chapter  Google Scholar 

  3. Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A.W.M., de Bruijne, M.: Early experiences with crowdsourcing airway annotations in chest CT. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 209–218. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46976-8_22

    Chapter  Google Scholar 

  4. Maier-Hein, L., Kondermann, D., Roß, T., Mersmann, S., Heim, E., Bodenstedt, S., Kenngott, H.G., Sanchez, A., Wagner, M., Preukschas, A.: Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int. J. Comput. Assist. Radiol. Surg. 10(8), 1201–1212 (2015)

    Article  Google Scholar 

  5. Guan, M.Y., Gulshan, V., Dai, A.M., Hinton, G.E.: Who said what: Modeling individual labelers improves classification. arXiv preprint arXiv:1703.08774 (2017)

  6. Codella, N.C., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). arXiv preprint arXiv:1710.05006 (2017)

  7. Abbasi, N.R., et al.: Early diagnosis of cutaneous melanoma: revisiting the abcd criteria. Jama 292(22), 2771–2776 (2004)

    Article  Google Scholar 

  8. Murthy, V., Hou, L., Samaras, D., Kurc, T.M., Saltz, J.H.: Center-focusing multi-task CNN with injected features for classification of glioma nuclear images. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 834–841. IEEE (2017)

    Google Scholar 

  9. Dhungel, N., Carneiro, G., Bradley, A.P.: A deep learning approach for the analysis of masses in mammograms with minimal user intervention. Med. Image Anal. 37, 114–128 (2017)

    Article  Google Scholar 

  10. Dumitrache, A., Aroyo, L., Welty, C.: Crowdsourcing ground truth for medical relation extraction. ACM Trans. Interact. Intell. Syst. (TiiS) 8(2), 12 (2018)

    Google Scholar 

Download references

Acknowledgments

We thank the students of the 8QA01 2017–2018 course for their participation in gathering the annotations.

Author information

Authors and Affiliations

  1. Medical Image Analysis, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands

    Veronika Cheplygina & Josien P. W. Pluim

  2. Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands

    Josien P. W. Pluim

Authors
  1. Veronika Cheplygina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Josien P. W. Pluim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Veronika Cheplygina .

Editor information

Editors and Affiliations

  1. University College London, London, UK

    Danail Stoyanov

  2. University of Leeds, Leeds, UK

    Zeike Taylor

  3. University of Barcelona, Barcelona, Spain

    Simone Balocco

  4. University of Bern, Bern, Switzerland

    Raphael Sznitman

  5. Sunnybrook Health Science Centre, Toronto, ON, Canada

    Anne Martel

  6. German Cancer Research Center, Heidelberg, Baden-Württemberg, Germany

    Lena Maier-Hein

  7. Ecole de Technologie Superieure, Montreal, QC, Canada

    Luc Duong

  8. Technical University of München, Garching, Bayern, Germany

    Guillaume Zahnd

  9. Technical University Munich, Munich, Germany

    Stefanie Demirci

  10. Technical University of München, Garching, Bayern, Germany

    Shadi Albarqouni

  11. Imperial College London, London, UK

    Su-Lin Lee

  12. University College London, London, UK

    Stefano Moriconi

  13. Eindhoven University of Technology, Eindhoven, The Netherlands

    Veronika Cheplygina

  14. Technical University of Munich, Garching, Germany

    Diana Mateus

  15. University of Dundee, Dundee, UK

    Emanuele Trucco

  16. École de Technologie Supérieure, Montréal, QC, Canada

    Eric Granger

  17. Universite de Rennes 1, Rennes, France

    Pierre Jannin

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cheplygina, V., Pluim, J.P.W. (2018). Crowd Disagreement About Medical Images Is Informative. In: Stoyanov, D., et al. Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. LABELS CVII STENT 2018 2018 2018. Lecture Notes in Computer Science(), vol 11043. Springer, Cham. https://doi.org/10.1007/978-3-030-01364-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01364-6_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01363-9

  • Online ISBN: 978-3-030-01364-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation