Crowdsourcing Labels for Pathological Patterns in CT Lung Scans: Can Non-experts Contribute Expert-Quality Ground Truth?

  • Alison Q. O’NeilEmail author
  • John T. Murchison
  • Edwin J. R. van Beek
  • Keith A. Goatman
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10552)


This paper investigates what quality of ground truth might be obtained when crowdsourcing specialist medical imaging ground truth from non-experts. Following basic tuition, 34 volunteer participants independently delineated regions belonging to 7 pathological patterns in 20 scans according to expert-provided pattern labels. Participants’ annotations were compared to a set of reference annotations using Dice similarity coefficient (DSC), and found to range between 0.41 and 0.77. The reference repeatability was 0.81. Analysis of prior imaging experience, annotation behaviour, scan ordering and time spent showed that only the last was correlated with annotation quality. Multiple observers combined by voxelwise majority vote outperformed a single observer, matching the reference repeatability for 5 of 7 patterns. In conclusion, crowdsourcing from non-experts yields acceptable quality ground truth, given sufficient expert task supervision and a sufficient number of observers per scan.



Many thanks to Phil Tolland who developed the software for the ground truth collection tool, and to all of the employees at Toshiba Medical Visualization Systems who took part in this study: Allan Barklie, Erin Beveridge, Antony Brown, Gerald Chau, Alasdair Corbett, Ross Davies, Matt Daykin, Ben Docherty, Venkatesh Gaddam, Keith Goatman, Marta Guarisco, Joseph Henry, Corné Hoogendoorn, Pia Kullik, Aneta Lisowska, Steve Magness, Craig Matear, James Matthews, Chris McGough, Haritha Miryala, Brian Mohr, Costas Plakas, Ian Poole, Marco Razeto, Faye Riley, Matt Shepherd, Simeon Skopalik, Andy Smout, Ken Sutherland, Paul Thomson, Phil Tolland, John Tough, Aidan Wellington and Gavin Wheeler.


  1. 1.
    Albarqouni, S., Matl, S., Baust, M., Navab, N., Demirci, S.: Playsourcing: a novel concept for knowledge creation in biomedical research. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 269–277. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_28 Google Scholar
  2. 2.
    Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., Mougiakakou, S.: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1207–1216 (2016)CrossRefGoogle Scholar
  3. 3.
    Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A.W.M., de Bruijne, M.: Early experiences with crowdsourcing airway annotations in chest CT. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 209–218. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_22 Google Scholar
  4. 4.
    Depeursinge, A., Vargas, A., Platon, A., Geissbuhler, A., Poletti, P.A., Müller, H.: Building a reference multimedia database for interstitial lung diseases. Comput. Med. Imaging Graph. 36(3), 227–238 (2012)CrossRefGoogle Scholar
  5. 5.
    Hansell, D.M., Bankier, A.A., MacMahon, H., McLoud, T.C., Müller, N.L., Remy, J.: Fleischner society: glossary of terms for thoracic imaging. Radiology 246(3), 697–722 (2008)CrossRefGoogle Scholar
  6. 6.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: Neural Information Processing Systems (2014)Google Scholar
  7. 7.
    Hossain, M., Kauranen, I.: Crowdsourcing: a comprehensive literature review. Strateg. Outsourcing Int. J. 8(1), 1753–8297 (2015)Google Scholar
  8. 8.
    Humphries, S.M., Yagihashi, K., Huckleberry, J., Rho, B.H., Schroeder, J.D., Strand, M., Schwarz, M.I., Flaherty, K.R., Kazerooni, E.A., van Beek, E.J.R., Lynch, D.A.: Idiopathic pulmonary fibrosis: data-driven textural analysis of extent of fibrosis at baseline and 15-month follow-up. Radiology 5, 161177 (2017)CrossRefGoogle Scholar
  9. 9.
    Langerak, T.R., van der Heide, U.A., Kotte, A.N., Viergever, M.A., van Vulpen, M., Pluim, J.P.: Label fusion in atlas-based segmentation using a selective and iterative method for performance level estimation (SIMPLE). IEEE Trans. Med. Imaging 29(12), 2000–2008 (2010)CrossRefGoogle Scholar
  10. 10.
    Van Leemput, K., Sabuncu, M.R.: A cautionary analysis of STAPLE using direct inference of segmentation truth. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 398–406. Springer, Cham (2014). doi: 10.1007/978-3-319-10404-1_50 Google Scholar
  11. 11.
    Luengo-Oroz, M.A., Arranz, A., Frean, J.: Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. J. Med. Internet Res. 14(6), e167 (2012)CrossRefGoogle Scholar
  12. 12.
    Piciucchi, S., Tomassetti, S., Ravaglia, C., Gurioli, C., Gurioli, C., Dubini, A., Carloni, A., Chilosi, M., Colby, T.V., Poletti, V.: From traction bronchiectasis to honeycombing in idiopathic pulmonary fibrosis: a spectrum of bronchiolar remodeling also in radiology? BMC Pulm. Med. 16(1), 87 (2016)CrossRefGoogle Scholar
  13. 13.
    Salisbury, M.L., Lynch, D.A., van Beek, E.J.R., Kazerooni, E.A., Guo, J., Xia, M., Murray, S., Anstrom, K.A., Yow, E., Martinez, F.J., Hoffman, E.A., Flaherty, K.R.: Idiopathic pulmonary fibrosis: the association between the adaptive multiple features method and fibrosis outcomes. Am. J. Respir. Crit. Care Med. 195(7), 921–929 (2017)CrossRefGoogle Scholar
  14. 14.
    Schlesinger, D., Jug, F., Myers, G., Rother, C., Kainmuller, D.: Crowdsourcing image segmentation with aSTAPLE. arXiv (2017)Google Scholar
  15. 15.
    Warfield, S.K., Zhou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans. Med. Imaging 23(7), 903–921 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Alison Q. O’Neil
    • 1
    Email author
  • John T. Murchison
    • 2
  • Edwin J. R. van Beek
    • 3
  • Keith A. Goatman
    • 1
  1. 1.Toshiba Medical Visualization Systems Ltd.EdinburghUK
  2. 2.Royal Infirmary of EdinburghEdinburghUK
  3. 3.Clinical Research Imaging CentreUniversity of EdinburghEdinburghUK

Personalised recommendations