Advertisement

Crowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence

  • L. Maier-Hein
  • T. Ross
  • J. Gröhl
  • B. Glocker
  • S. Bodenstedt
  • C. Stock
  • E. Heim
  • M. Götz
  • S. Wirkert
  • H. Kenngott
  • S. Speidel
  • K. Maier-Hein
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9901)

Abstract

With the recent breakthrough success of machine learning based solutions for automatic image annotation, the availability of reference image annotations for algorithm training is one of the major bottlenecks in medical image segmentation and many other fields. Crowdsourcing has evolved as a valuable option for annotating large amounts of data while sparing the resources of experts, yet, segmentation of objects from scratch is relatively time-consuming and typically requires an initialization of the contour. The purpose of this paper is to investigate whether the concept of crowd-algorithm collaboration can be used to simultaneously (1) speed up crowd annotation and (2) improve algorithm performance based on the feedback of the crowd. Our contribution in this context is two-fold: Using benchmarking data from the MICCAI 2015 endoscopic vision challenge we show that atlas forests extended by a novel superpixel-based confidence measure are well-suited for medical instrument segmentation in laparoscopic video data. We further demonstrate that the new algorithm and the crowd can mutually benefit from each other in a collaborative annotation process. Our method can be adapted to various applications and thus holds high potential to be used for large-scale low-cost data annotation.

References

  1. 1.
    Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 35, 1313–1321 (2016)CrossRefGoogle Scholar
  2. 2.
    Allan, M., Chang, P.-L., Ourselin, S., Hawkes, D.J., Sridhar, A., Kelly, J., Stoyanov, D.: Image based surgical instrument pose estimation with multi-class labelling and optical flow. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 331–338. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_41 CrossRefGoogle Scholar
  3. 3.
    Bodenstedt, S., Goertler, J., Wagner, M., Kenngott, H., Mueller-Stich, B.P., Dillmann, R., Speidel, S.: Superpixel-based structure classification for laparoscopic surgery. In: SPIE Medical Imaging, p. 978618 (2016)Google Scholar
  4. 4.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). http://dx.doi.org/10.1023/A%3A1010933404324
  5. 5.
    Cheng, J., Bernstein, M.S.: Flock: hybrid crowd-machine learning classifiers. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 600–611. ACM (2015)Google Scholar
  6. 6.
    Maier-Hein, L., et al.: Can masses of non-experts train highly accurate image classifiers? In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 438–445. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10470-6_55 Google Scholar
  7. 7.
    Radu, A.-L., Ionescu, B., Menéndez, M., Stöttinger, J., Giunchiglia, F., Angeli, A.: A hybrid machine-crowd approach to photo retrieval result diversification. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MMM 2014. LNCS, vol. 8325, pp. 25–36. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-04114-8_3 CrossRefGoogle Scholar
  8. 8.
    Zikic, D., Glocker, B., Criminisi, A.: Classifier-based multi-atlas label propagation with test-specific atlas weighting for correspondence-free scenarios. In: Menze, B., Langs, G., Montillo, A., Kelm, M., Müller, H., Zhang, S., Cai, W.T., Metaxas, D. (eds.) MCV 2014. LNCS, vol. 8848, pp. 116–124. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-13972-2_11 Google Scholar
  9. 9.
    Zikic, D., Glocker, B., Criminisi, A.: Encoding atlases by randomized classification forests for efficient multi-atlas label propagation. Med. Image Anal. 18(8), 1262–1273 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • L. Maier-Hein
    • 1
  • T. Ross
    • 1
  • J. Gröhl
    • 1
  • B. Glocker
    • 2
  • S. Bodenstedt
    • 3
  • C. Stock
    • 4
  • E. Heim
    • 1
  • M. Götz
    • 5
  • S. Wirkert
    • 1
  • H. Kenngott
    • 6
  • S. Speidel
    • 3
  • K. Maier-Hein
    • 5
  1. 1.Computer-assisted Interventions GroupGerman Cancer Research Center (DKFZ)HeidelbergGermany
  2. 2.Biomedical Image Analysis GroupImperial College LondonLondonUK
  3. 3.Institute for Anthropomatics and RoboticsKarlsruhe Institute of TechnologyKarlsruheGermany
  4. 4.Institute of Medical Biometry and InformaticsUniversity of HeidelbergHeidelbergGermany
  5. 5.Medical Image Computing GroupDKFZHeidelbergGermany
  6. 6.Department of General, Visceral and Transplant SurgeryUniversity of HeidelbergHeidelbergGermany

Personalised recommendations