With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm.
So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images.
On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %).
Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Ashburner J (2012) SPM: a history. Neuroimage 62(2):791–800
Brabham DC (2008) Crowdsourcing as a model for problem solving an introduction and cases. Converg Int J Res New Media Technol 14(1):75–90
Brady CJ, Villanti AC, Pearson JL, Kirchner TR, Gupta OP, Shah CP (2014) Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res 16(10):e233
Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187(1):65–71
Dale AM, Fischl B, Sereno MI (1999) Cortical surface-based analysis: I. Segmentation and surface reconstruction. Neuroimage 9(2):179–194
Fischl B (2012) FreeSurfer. Neuroimage 62(2):774–781
Fischl B, Sereno MI, Dale AM (1999) Cortical surface-based analysis: II. Inflation, flattening, and a surface-based coordinate system. Neuroimage 9(2):195–207
Foncubierta Rodríguez A, Müller H (2012) Ground truth generation in medical imaging: a crowdsourcing-based iterative approach. In: Proceedings of the ACM multimedia workshop on crowdsourcing for multimedia. ACM, pp 9–14
Fritzsche KH, Neher PF, Reicht I, van Bruggen T, Goch C, Reisert M, Nolden M, Zelzer S, Meinzer HP, Stieltjes B (2012) MITK diffusion imaging. Methods Inf Med 51(5):441
Haahr M, Hansen D, Fisher P, Svarer C, Stenbæk D, Madsen K, Madsen J, Holst JJ, Baaré W, Hojgaard L, Almdal T, Knudsen G (2015) Central 5-HT neurotransmission modulates weight loss following gastric bypass surgery in obese individuals. J Neurosci 35(14):5884–5889
de Herrera AGS, Foncubierta-Rodríguez A, Markonis D, Schaer R, Müller H (2014) Crowdsourcing for medical image classification. Swiss Med Inform 30
Jannin P, Grova C, Maurer CR Jr (2006) Model for defining and reporting reference-based validation protocols in medical image processing. Int J Comput Assist Radiol Surg 1(2):63–73
Jenkinson M, Beckmann CF, Behrens TE, Woolrich MW, Smith SM (2012) FSL. Neuroimage 62(2):782–790
Maier-Hein L, Kondermann D, Roß T, Mersmann S, Heim E, Bodenstedt S, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, März K, Mehrabi A, Speidel S, Stock C (2015) Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int J Comput Assist Radiol Surg 10(8):1201–1212
Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, Speidel S (2014) Can masses of non-experts train highly accurate image classifiers? In: Medical image computing and computer-assisted intervention (MICCAI 2014). Springer, New York, pp 438–445
Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, Bodenstedt S, Speidel S (2014) Crowdsourcing for reference correspondence generation in endoscopic images. In: Medical image computing and computer-assisted intervention (MICCAI 2014). Springer, New York, pp 349–356
Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, Padmanabhan S, Nielsen K, Ozcan A (2012) Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One 7(5):37245
Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, Burns JE, Summers RM (2012) Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 262(3):824–833
Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, Becker LB, Seymour AK, Merchant RM (2014) Crowdsourcing harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med 29(1):187–203
Collection of data that were included in the study was supported by the Lundbeck Foundation Center of Excellence Cimbi (R90-A7722). Melanie Ganz’ research was supported by the Carlsberg Foundation (2013-01-0502) and National Institutes of Health (NIH) (5R21EB018964-02). The work was further conducted within the setting of SFB TRR 125: cognition-guided surgery funded by the German Research Foundation (DFG) (Project A02).
Conflict of interest
Melanie Ganz, Daniel Kondermann, Jonas Andrulis and Lena Maier-Hein declare that they have no conflict of interest. Gitte Moos Knudsen has been an invited lecturer at Pfizer A/S, worked as a consultant and received grants from H. Lundbeck A/S and is a stockholder of Novo Nordisk/Novozymes. Furthermore, she is on the board of directors of the BrainPrize and Elsass Foundation and the Advisory Board of the Kristian Jebsen Foundation and has authored for FADL and served as editor for Elsevier (IJNP).
All procedures followed were in accordance with the ethical standards of the responsible committees on human experimentation (institutional and national) and with the Declaration of Helsinki 1975, as revised in 2008. Informed consent was obtained from all patients for being included in the study.
About this article
Cite this article
Ganz, M., Kondermann, D., Andrulis, J. et al. Crowdsourcing for error detection in cortical surface delineations. Int J CARS 12, 161–166 (2017). https://doi.org/10.1007/s11548-016-1445-9
- Cortical surface