Early Experiences with Crowdsourcing Airway Annotations in Chest CT

  • Veronika Cheplygina
  • Adria Perez-Rovira
  • Wieying Kuo
  • Harm A. W. M. Tiddens
  • Marleen de Bruijne
Conference paper

DOI: 10.1007/978-3-319-46976-8_22

Part of the Lecture Notes in Computer Science book series (LNCS, volume 10008)
Cite this paper as:
Cheplygina V., Perez-Rovira A., Kuo W., Tiddens H.A.W.M., de Bruijne M. (2016) Early Experiences with Crowdsourcing Airway Annotations in Chest CT. In: Carneiro G. et al. (eds) Deep Learning and Data Labeling for Medical Applications. LABELS 2016, DLMIA 2016. Lecture Notes in Computer Science, vol 10008. Springer, Cham

Abstract

Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Veronika Cheplygina
    • 1
    • 2
  • Adria Perez-Rovira
    • 1
    • 3
  • Wieying Kuo
    • 3
    • 4
  • Harm A. W. M. Tiddens
    • 3
    • 4
  • Marleen de Bruijne
    • 1
    • 5
  1. 1.Biomedical Imaging Group Rotterdam, Departments Medical Informatics and RadiologyErasmus Medical CenterRotterdamThe Netherlands
  2. 2.Pattern Recognition LaboratoryDelft University of TechnologyDelftThe Netherlands
  3. 3.Department of Pediatric Pulmonology and AllergologyErasmus Medical Center - Sophia Children’s HospitalRotterdamThe Netherlands
  4. 4.Department of RadiologyErasmus Medical CenterRotterdamThe Netherlands
  5. 5.Image Section, Departments of Computer ScienceUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations