Advertisement

Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences

  • Lena Maier-Hein
  • Daniel Kondermann
  • Tobias Roß
  • Sven Mersmann
  • Eric Heim
  • Sebastian Bodenstedt
  • Hannes Götz Kenngott
  • Alexandro Sanchez
  • Martin Wagner
  • Anas Preukschas
  • Anna-Laura Wekerle
  • Stefanie Helfert
  • Keno März
  • Arianeb Mehrabi
  • Stefanie Speidel
  • Christian Stock
Original Article

Abstract

Purpose

Feature tracking and 3D surface reconstruction are key enabling techniques to computer-assisted minimally invasive surgery. One of the major bottlenecks related to training and validation of new algorithms is the lack of large amounts of annotated images that fully capture the wide range of anatomical/scene variance in clinical practice. To address this issue, we propose a novel approach to obtaining large numbers of high-quality reference image annotations at low cost in an extremely short period of time.

Methods

The concept is based on outsourcing the correspondence search to a crowd of anonymous users from an online community (crowdsourcing) and comprises four stages: (1) feature detection, (2) correspondence search via crowdsourcing, (3) merging multiple annotations per feature by fitting Gaussian finite mixture models, (4) outlier removal using the result of the clustering as input for a second annotation task.

Results

On average, 10,000 annotations were obtained within 24 h at a cost of $100. The annotation of the crowd after clustering and before outlier removal was of expert quality with a median distance of about 1 pixel to a publically available reference annotation. The threshold for the outlier removal task directly determines the maximum annotation error, but also the number of points removed.

Conclusions

Our concept is a novel and effective method for fast, low-cost and highly accurate correspondence generation that could be adapted to various other applications related to large-scale data annotation in medical image computing and computer-assisted interventions.

Keywords

Crowdsourcing Validation Benchmarking Endoscopy Image correspondences Feature tracking 

Notes

Acknowledgments

This work was conducted within the setting of SFB TRR 125: Cognition-guided surgery funded by the German Research Foundation (DFG) (Projects A02 and A01). It was further sponsored by the European Social Fund of the State of Baden-Württemberg and the Klaus Tschira Foundation.

References

  1. 1.
    Maier-Hein L, Groch A, Bartoli A, Bodenstedt S, Boissonnat G, Chang PL, Clancy NT, Elson DS, Haase S, Heim E, Hornegger J, Jannin P, Kenngott H, Kilgus T, Muller-Stich B, Oladokun D, Rohl S, Dos Santos TR, Schlemmer HP, Seitel A, Speidel S, Wagner M, Stoyanov D (2014) Comparative validation of single-shot optical techniques for laparoscopic 3-D surface reconstruction. IEEE Trans Med Imaging 33:1913–1930PubMedCrossRefGoogle Scholar
  2. 2.
    Von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 319–326Google Scholar
  3. 3.
    Estellés-Arolas E, González-Ladrón-de Guevara F (2012) Towards an integrated crowdsourcing definition. J Inf Sci 38:189–200CrossRefGoogle Scholar
  4. 4.
    Chen JJ, Menezes NJ, Bradley AD, North T (2011) Opportunities for crowdsourcing research on amazon mechanical turk. Interfaces 5. Jg., No. 3Google Scholar
  5. 5.
    Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173CrossRefGoogle Scholar
  6. 6.
    Khatib F, DiMaio F, Cooper S, Kazmierczyk M, Gilski M, Krzywda S, Zabranska H, Pichova I, Thompson J, Popovic Z, Jaskolski M, Baker D (2011) Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol 18:1175–1177Google Scholar
  7. 7.
    Ranard B, Ha Y, Meisel Z, Asch D, Hill S, Becker L, Seymour A, Merchant R (2014) Crowdsourcing—harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med 29:187–203PubMedCentralPubMedCrossRefGoogle Scholar
  8. 8.
    Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, Padmanabhan S, Nielsen K, Ozcan A (2012) Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS ONE 7:e37245PubMedCentralPubMedCrossRefGoogle Scholar
  9. 9.
    Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, Burns JE, Summers RM (2012) Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 262:824–833PubMedCentralPubMedCrossRefGoogle Scholar
  10. 10.
    Foncubierta Rodríguez A, Müller H (2012) Ground truth generation in medical imaging: a crowdsourcing-based iterative approach. In: Proceedings of the ACM multimedia 2012 workshop on crowdsourcing for multimedia. CrowdMM’12. ACM, New York, NY, USA, pp 9–14Google Scholar
  11. 11.
    Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187:65–71PubMedCrossRefGoogle Scholar
  12. 12.
    Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, Speidel S (2014) Can masses of non-experts train highly accurate image classifiers? In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8674. Lecture notes in computer science. Springer, pp 438–445Google Scholar
  13. 13.
    Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, Bodenstedt S, Speidel S (2014) Crowdsourcing for reference correspondence generation in endoscopic images. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8674. Lecture notes in computer science. Springer, pp 349–356Google Scholar
  14. 14.
    Bay H, Tuytelaars T, Van Gool L (2006) SURF: speeded up robust features. In: European conference on computer vision (ECCV), vol 3951. Lecture notes in computer science, pp 404–417Google Scholar
  15. 15.
    Morris RR, Dontcheva M, Gerber EM (2012) Priming for better performance in microtask crowdsourcing environments. IEEE Internet Comput 16:13–19CrossRefGoogle Scholar
  16. 16.
    Fraley C, Raftery AE (2002) Model-based clustering, discriminant analysis and density estimation. J Am Stat Assoc 97:611–631CrossRefGoogle Scholar
  17. 17.
    Fraley C, Raftery AE, Murphy TB, Scrucca L (2012) MCLUST version 4 for R: normal mixture modeling for model-based clustering, classification, and density estimation. Technical report, Technical report no. 597, Department of Statistics, University of WashingtonGoogle Scholar
  18. 18.
    Puerto G, Mariottini GL (2012) A comparative study of correspondence-search algorithms in MIS images. In: International conference on medical image computing and computer-assisted intervention (MICCAI), pp 625–633Google Scholar
  19. 19.
    R Core Team (2014) R: A language and environment for statistical computing. R Core Team, ViennaGoogle Scholar
  20. 20.
    Pinheiro J, Bates D, DebRoy S, Sarkar D (2014) R Core Team: NLME: linear and nonlinear mixed effects models. R package version 3.1-118Google Scholar

Copyright information

© CARS 2015

Authors and Affiliations

  • Lena Maier-Hein
    • 1
  • Daniel Kondermann
    • 2
  • Tobias Roß
    • 1
  • Sven Mersmann
    • 1
  • Eric Heim
    • 1
  • Sebastian Bodenstedt
    • 3
  • Hannes Götz Kenngott
    • 4
  • Alexandro Sanchez
    • 2
  • Martin Wagner
    • 4
  • Anas Preukschas
    • 4
  • Anna-Laura Wekerle
    • 4
  • Stefanie Helfert
    • 4
  • Keno März
    • 1
  • Arianeb Mehrabi
    • 4
  • Stefanie Speidel
    • 3
  • Christian Stock
    • 5
  1. 1.Junior Group Computer-Assisted InterventionsGerman Cancer Research Center (DKFZ)HeidelbergGermany
  2. 2.Heidelberg Collaboratory for Image ProcessingUniversity of HeidelbergHeidelbergGermany
  3. 3.Institute for Anthropomatics, Karlsruhe Institute of Technology (KIT)KarlsruheGermany
  4. 4.Department of General, Visceral and Transplant SurgeryUniversity of HeidelbergHeidelbergGermany
  5. 5.Institute of Medical Biometry and InformaticsUniversity of HeidelbergHeidelbergGermany

Personalised recommendations