Role of Task Complexity and Training in Crowdsourced Image Annotation

  • Nadine S. Schaadt
  • Anne Grote
  • Germain Forestier
  • Cédric Wemmert
  • Friedrich FeuerhakeEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11039)


Accurate annotation of anatomical structures or pathological changes in microscopic images is an important task in computational pathology. Crowdsourcing holds promise to address this demand, but so far feasibility has only be shown for simple tasks and not for high-quality annotation of complex structures which is often limited by shortage of experts. Third-year medical students participated in solving two complex tasks, labeling of images and delineation of relevant image objects in breast cancer and kidney tissue. We evaluated their performance and addressed the requirements of task complexity and training phases. Our results show feasibility and a high agreement between students and experts. The training phase improved accuracy of image labeling.


Crowdsourcing Human decision making Image classification Image delineation Digital pathology Annotation 



We thank all students for contribution; M. Temerinac-Ott, Icube; R. Schönmeyer, C. Vanegas, Definiens for help in data selection; G. Stiller, M. Behrends, Peter L. Reichertz Institute for Medical Informatics; and A.-K. Rieke for the video.


  1. 1.
    Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imag. 35, 1313–1321 (2016)CrossRefGoogle Scholar
  2. 2.
    Della Mea, V., Maddalena, E., Mizzaro, S., Machin, P., Beltrami, C.A.: Preliminary results from a crowdsourcing experiment in immunohistochemistry. Diagn. Pathol. 9, S6 (2014)CrossRefGoogle Scholar
  3. 3.
    Hoßfeld, T., et al.: Best practices and recommendations for crowdsourced qoe-lessons learned from the qualinet task force crowdsourcing. In: QUALINET (2014)Google Scholar
  4. 4.
    Irshad, H., et al.: Crowdsourcing scoring of immunohistochemistry images: evaluating performance of the crowd and an automated computational method. Sci. Rep. 7 (2017)Google Scholar
  5. 5.
    Kim, E., Mente, S., Keenan, A., Gehlot, V.: Digital pathology annotation data for improved deep neural network classification. In: SPIE Medical Imaging, p. 101380D (2017)Google Scholar
  6. 6.
    Lawson, J., et al.: Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays. Br. J. Cancer 116, 237–245 (2017)CrossRefGoogle Scholar
  7. 7.
    Liu, S., Xia, F., Zhang, J., Wang, L., Wang, L.: How crowdsourcing risks affect performance: an exploratory model. Manag. Decis. 54, 2235–2255 (2016)CrossRefGoogle Scholar
  8. 8.
    Marée, R.: Collaborative analysis of multi-gigapixel imaging data using cytomine. Bioinformatics 32, 1395–1401 (2016)CrossRefGoogle Scholar
  9. 9.
    Mavandadi, S., et al.: Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PloS One 7, e37245 (2012)CrossRefGoogle Scholar
  10. 10.
    Redi, J., Povoa, I.: Crowdsourcing for rating image aesthetic appeal: better a paid or a volunteer crowd? In: Proceedings of 2014 International ACM Workshop Crowdsourcing Multimedia, pp. 25–30. ACM (2014)Google Scholar
  11. 11.
    dos Reis, F.J.C., et al.: Crowdsourcing the general public for large scale molecular pathology studies in cancer. EBioMedicine 2, 681–689 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Nadine S. Schaadt
    • 1
  • Anne Grote
    • 1
  • Germain Forestier
    • 2
  • Cédric Wemmert
    • 3
  • Friedrich Feuerhake
    • 1
    • 4
    Email author
  1. 1.Institute for PathologyHannover Medical SchoolHannoverGermany
  2. 2.IRIMAS, University of Haute AlsaceMulhouseFrance
  3. 3.ICube, University of StrasbourgIllkirchFrance
  4. 4.Institute for NeuropathologyUniversity Clinic FreiburgFreiburg im BreisgauGermany

Personalised recommendations