Skip to main content
Book cover

International Workshop on Deep Learning in Medical Image Analysis

International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis

DLMIA 2016, LABELS 2016: Deep Learning and Data Labeling for Medical Applications pp 269–277Cite as

Playsourcing: A Novel Concept for Knowledge Creation in Biomedical Research

Part of the Lecture Notes in Computer Science book series (LNIP,volume 10008)

Abstract

Being considered as a valid solution to the lack of ground truth data problem, crowdsourcing has recently gained a lot of attention within the biomedical domain. However, available concepts in life science domain require expert knowledge and thereby restrict the access to only very specific communities. In this paper, we go beyond state-of-the-art and present a novel concept for seamlessly embedding biomedical science into a common game canvas. Besides introducing the visual saliency concept, we thereby essentially eliminate the requirement for prior knowledge. We have further implemented a game to evaluate our novel concept in three different user studies.

Keywords

  • Gamification
  • Crowdsourcing
  • Aggregation
  • Annotation

S. Albarqouni and S. Matl—contributed equally towards this work.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-46976-8_28
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-46976-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

Notes

  1. 1.

    http://amida13.isi.uu.nl/.

References

  1. Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 35(5), 1313–1321 (2016)

    CrossRef  Google Scholar 

  2. Aroyo, L., Welty, C.: The three sides of crowdtruth. Hum. Comput. 1(1), 31–44 (2014)

    CrossRef  Google Scholar 

  3. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z.: Foldit players: predicting protein structures with a multiplayer online game. Nature 446, 756–760 (2010)

    CrossRef  Google Scholar 

  4. Dumitrache, A., Aroyo, L., Welty, C., Sips, R.J., Levas, A.: Dr. detective: combining gamication techniques and crowdsourcing to create a gold standard in medical text. In: Proceedings of the 1st International Conference on Crowdsourcing the Semantic Web-Volume 1030, pp. 16–31. CEUR-WS. org (2013)

    Google Scholar 

  5. Fuchs, J., Isenberg, P., Bezerianos, A., Fischer, F., Bertini, E.: The influence of contour on similarity perception of star glyphs. IEEE Trans. Vis. Comput. Graph. 20(12), 2251–2260 (2014)

    CrossRef  Google Scholar 

  6. Good, B.M., Loguercio, S., Griffith, O.L., Nanis, M., Wu, C., Su, A.I.: The cure: design and evaluation of a crowdsourcing game for gene selection for breast cancer survival prediction. JMIR Serious Games 2(2), e7 (2014). doi:10.2196/games

    CrossRef  Google Scholar 

  7. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work?-a literature review of empirical studies on gamification. In: 2014 47th Hawaii International Conference on System Sciences (HICSS), pp. 3025–3034. IEEE (2014)

    Google Scholar 

  8. Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. Inf. Comput. 108(2), 212–261 (1994)

    MathSciNet  CrossRef  MATH  Google Scholar 

  9. Luengo-Oroz, M.A., Arranz, A., Frean, J.: Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. J. Med. Internet Res. 14(6), e167 (2012)

    CrossRef  Google Scholar 

  10. Macenko, M., Niethammer, M., Marron, J., Borland, D., Woosley, J.T., Guan, X., Schmitt, C., Thomas, N.E.: A method for normalizing histology slides for quantitative analysis. In: ISBI, vol. 9, pp. 1107–1110 (2009)

    Google Scholar 

  11. Maier-Hein, L., Ross, T., Glocker, B., Bodenstedt, S., Stock, C., Heim, E., Wirkert, S., Kenngott, H., Speidel, S., Maier-Hein, K., et al.: Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence

    Google Scholar 

  12. Maier-Hein, L., Kondermann, D., Roß, T., Mersmann, S., Heim, E., Bodenstedt, S., Kenngott, H.G., Sanchez, A., Wagner, M., Preukschas, A., Wekerle, A.L., Helfert, S., März, K., Mehrabi, A., Speidel, S., Stock, C.: Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int. J. Comput. Assist. Radiol. Surg. 10(8), 1201–1212 (2015)

    CrossRef  Google Scholar 

  13. Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11, 19–60 (2010)

    MathSciNet  MATH  Google Scholar 

  14. Veta, M., Van Diest, P.J., Willems, S.M., Wang, H., Madabhushi, A., Cruz-Roa, A., Gonzalez, F., Larsen, A.B., Vestergaard, J.S., Dahl, A.B., et al.: Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med. Image Anal. 20(1), 237–248 (2015)

    CrossRef  Google Scholar 

  15. Wertheimer, M.: Untersuchungen zur lehre von der gestalt. ii. Psychologische Forschung 4(1), 301–350 (1923). http://dx.doi.org/10.1007/BF00410640

    CrossRef  Google Scholar 

  16. Zheng, Y., Liu, D., Georgescu, B., Nguyen, H., Comaniciu, D.: 3D deep learning for efficient and robust landmark detection in volumetric data. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 565–572. Springer, Heidelberg (2015). doi:10.1007/978-3-319-24553-9_69

    CrossRef  Google Scholar 

Download references

Acknowledgment

We would like to thank all anonymous players who participate in our game. We are also grateful to Dr. Mitko Veta for giving us the permission to use the AMIDA13 dataset in our research.

Author information

Authors and Affiliations

  1. Chair for Computer Aided Medical Procedure (CAMP), Technische Universität München (TUM), Munich, Germany

    Shadi Albarqouni, Stefan Matl, Maximilian Baust, Nassir Navab & Stefanie Demirci

  2. Johns Hopkins University, Baltimore, Maryland, USA

    Nassir Navab

Authors
  1. Shadi Albarqouni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Matl
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maximilian Baust
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nassir Navab
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stefanie Demirci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shadi Albarqouni .

Editor information

Editors and Affiliations

  1. University of Adelaide, Adelaide, South Australia, Australia

    Gustavo Carneiro

  2. Technical University of Munich, Garching, Germany

    Diana Mateus

  3. Technical University of Munich, Garching, Germany

    Loïc Peter

  4. University of Queensland, St Lucia, Queensland, Australia

    Andrew Bradley

  5. Universidade do Porto, Porto, Portugal

    João Manuel R. S. Tavares

  6. University of Oxford, Oxford, United Kingdom

    Vasileios Belagiannis

  7. Universidade Estadual Paulista, Bauru, Brazil

    João Paulo Papa

  8. Instituto Superior Técnico, Lisbon, Portugal

    Jacinto C. Nascimento

  9. Delft University of Technology, Delft, The Netherlands

    Marco Loog

  10. University of South Australia, Adelaide, South Australia, Australia

    Zhi Lu

  11. Universidade do Porto, Porto, Portugal

    Jaime S. Cardoso

  12. Google DeepMind, London, United Kingdom

    Julien Cornebise

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Albarqouni, S., Matl, S., Baust, M., Navab, N., Demirci, S. (2016). Playsourcing: A Novel Concept for Knowledge Creation in Biomedical Research. In: , et al. Deep Learning and Data Labeling for Medical Applications. DLMIA LABELS 2016 2016. Lecture Notes in Computer Science(), vol 10008. Springer, Cham. https://doi.org/10.1007/978-3-319-46976-8_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-46976-8_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46975-1

  • Online ISBN: 978-3-319-46976-8

  • eBook Packages: Computer ScienceComputer Science (R0)