Early Experiences with Crowdsourcing Airway Annotations in Chest CT

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10008)


Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.


Airway Wall Usable Annotation Airway Lumen Single Ellipse Pediatric Compute Tomography 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was partially funded by the research project “Transfer learning in biomedical image analysis” which is financed by the Netherlands Organization for Scientific Research (NWO) grant no. 639.022.010. We gratefully acknowledge Dr. Daniel Kondermann of Heidelberg University for his help with crowdsourcing, and the anonymous reviewers for their constructive comments.


  1. 1.
    Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 35(5), 1313–1321 (2016)CrossRefGoogle Scholar
  2. 2.
    Chen, J.J., Menezes, N.J., Bradley, A.D., North, T.A.: Opportunitiesfor crowdsourcing research on Amazon Mechanical Turk. In: CHI workshop on Crowdsourcing and Human Computation (2011)Google Scholar
  3. 3.
    Kuo, W., et al.: Assessment of bronchiectasis in children with cystic fibrosis by comparing airway and artery dimensions to normal controls on inspiratory and expiratory spirometer guided chest computed tomography. In: ECR 2015-European Congress of Radiology (2015)Google Scholar
  4. 4.
    Lo, P., Sporring, J., Ashraf, H., Pedersen, J.J., Bruijne, M.: Vessel-guided airway tree segmentation: a voxel classification approach. Med. Image Anal. 14(4), 527–538 (2010)CrossRefGoogle Scholar
  5. 5.
    Maier-Hein, L., Kondermann, D., et al.: Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int. J. Comput. Assist. Radiol. Surg. 10(8), 1201–1212 (2015)CrossRefGoogle Scholar
  6. 6.
    Mitry, D., Peto, T., Hayat, S., Blows, P., Morgan, J., Khaw, K.T., Foster, P.J.: Crowdsourcing as a screening tool to detect clinical features of glaucomatous optic neuropathy from digital photography. PLoS ONE 10(2), e0117401 (2015)CrossRefGoogle Scholar
  7. 7.
    Mott, L.S., Graniel, K.G., Park, J., Klerk, N.H., Sly, P.D., Murray, C.P., Tiddens, H.A.W.M., Stick, S.M.: Assessment of early bronchiectasis in young children with cystic fibrosis is dependent on lung volume. CHEST J. 144(4), 1193–1198 (2013)CrossRefGoogle Scholar
  8. 8.
    Nguyen, T.B., Wang, S., Anugu, V., Rose, N., McKenna, M., Petrick, N., Burns, J.E., Summers, R.M.: Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 262(3), 824–833 (2012)CrossRefGoogle Scholar
  9. 9.
    Perez-Rovira, A., Kuo, W., Petersen, J., Tiddens, H., de Bruijne, M.: Automated quantification of bronchiectasis, airway wall thickening and lumen tapering in chest CT. In: ECR 2015-European Congress of Radiology (2015)Google Scholar
  10. 10.
    Petersen, J., Nielsen, M., Lo, P., Nordenmark, L.H., Pedersen, J.H., Wille, M.M.W., Dirksen, A., de Bruijne, M.: Optimal surface segmentation using flow lines to quantify airway abnormalities in chronic obstructive pulmonary disease. Med. Image Anal. 18(3), 531–541 (2014)CrossRefGoogle Scholar
  11. 11.
    Tiddens, H.A.W.M., Donaldson, S.H., Rosenfeld, M., Paré, P.D.: Cystic fibrosis lung disease starts in the small airways: can we treat it more effectively? Pediatr. Pulmonol. 45(2), 107–117 (2010)CrossRefGoogle Scholar
  12. 12.
    World Health Organization: Fact sheet nr 10. Online (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Biomedical Imaging Group Rotterdam, Departments Medical Informatics and RadiologyErasmus Medical CenterRotterdamThe Netherlands
  2. 2.Pattern Recognition LaboratoryDelft University of TechnologyDelftThe Netherlands
  3. 3.Department of Pediatric Pulmonology and AllergologyErasmus Medical Center - Sophia Children’s HospitalRotterdamThe Netherlands
  4. 4.Department of RadiologyErasmus Medical CenterRotterdamThe Netherlands
  5. 5.Image Section, Departments of Computer ScienceUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations