Skip to main content

Playsourcing: A Novel Concept for Knowledge Creation in Biomedical Research

Part of the Lecture Notes in Computer Science book series (LNIP,volume 10008)


Being considered as a valid solution to the lack of ground truth data problem, crowdsourcing has recently gained a lot of attention within the biomedical domain. However, available concepts in life science domain require expert knowledge and thereby restrict the access to only very specific communities. In this paper, we go beyond state-of-the-art and present a novel concept for seamlessly embedding biomedical science into a common game canvas. Besides introducing the visual saliency concept, we thereby essentially eliminate the requirement for prior knowledge. We have further implemented a game to evaluate our novel concept in three different user studies.


  • Gamification
  • Crowdsourcing
  • Aggregation
  • Annotation

S. Albarqouni and S. Matl—contributed equally towards this work.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-46976-8_28
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-46976-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.


  1. 1.


  1. Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 35(5), 1313–1321 (2016)

    CrossRef  Google Scholar 

  2. Aroyo, L., Welty, C.: The three sides of crowdtruth. Hum. Comput. 1(1), 31–44 (2014)

    CrossRef  Google Scholar 

  3. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z.: Foldit players: predicting protein structures with a multiplayer online game. Nature 446, 756–760 (2010)

    CrossRef  Google Scholar 

  4. Dumitrache, A., Aroyo, L., Welty, C., Sips, R.J., Levas, A.: Dr. detective: combining gamication techniques and crowdsourcing to create a gold standard in medical text. In: Proceedings of the 1st International Conference on Crowdsourcing the Semantic Web-Volume 1030, pp. 16–31. CEUR-WS. org (2013)

    Google Scholar 

  5. Fuchs, J., Isenberg, P., Bezerianos, A., Fischer, F., Bertini, E.: The influence of contour on similarity perception of star glyphs. IEEE Trans. Vis. Comput. Graph. 20(12), 2251–2260 (2014)

    CrossRef  Google Scholar 

  6. Good, B.M., Loguercio, S., Griffith, O.L., Nanis, M., Wu, C., Su, A.I.: The cure: design and evaluation of a crowdsourcing game for gene selection for breast cancer survival prediction. JMIR Serious Games 2(2), e7 (2014). doi:10.2196/games

    CrossRef  Google Scholar 

  7. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work?-a literature review of empirical studies on gamification. In: 2014 47th Hawaii International Conference on System Sciences (HICSS), pp. 3025–3034. IEEE (2014)

    Google Scholar 

  8. Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. Inf. Comput. 108(2), 212–261 (1994)

    MathSciNet  CrossRef  MATH  Google Scholar 

  9. Luengo-Oroz, M.A., Arranz, A., Frean, J.: Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. J. Med. Internet Res. 14(6), e167 (2012)

    CrossRef  Google Scholar 

  10. Macenko, M., Niethammer, M., Marron, J., Borland, D., Woosley, J.T., Guan, X., Schmitt, C., Thomas, N.E.: A method for normalizing histology slides for quantitative analysis. In: ISBI, vol. 9, pp. 1107–1110 (2009)

    Google Scholar 

  11. Maier-Hein, L., Ross, T., Glocker, B., Bodenstedt, S., Stock, C., Heim, E., Wirkert, S., Kenngott, H., Speidel, S., Maier-Hein, K., et al.: Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence

    Google Scholar 

  12. Maier-Hein, L., Kondermann, D., Roß, T., Mersmann, S., Heim, E., Bodenstedt, S., Kenngott, H.G., Sanchez, A., Wagner, M., Preukschas, A., Wekerle, A.L., Helfert, S., März, K., Mehrabi, A., Speidel, S., Stock, C.: Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int. J. Comput. Assist. Radiol. Surg. 10(8), 1201–1212 (2015)

    CrossRef  Google Scholar 

  13. Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11, 19–60 (2010)

    MathSciNet  MATH  Google Scholar 

  14. Veta, M., Van Diest, P.J., Willems, S.M., Wang, H., Madabhushi, A., Cruz-Roa, A., Gonzalez, F., Larsen, A.B., Vestergaard, J.S., Dahl, A.B., et al.: Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med. Image Anal. 20(1), 237–248 (2015)

    CrossRef  Google Scholar 

  15. Wertheimer, M.: Untersuchungen zur lehre von der gestalt. ii. Psychologische Forschung 4(1), 301–350 (1923).

    CrossRef  Google Scholar 

  16. Zheng, Y., Liu, D., Georgescu, B., Nguyen, H., Comaniciu, D.: 3D deep learning for efficient and robust landmark detection in volumetric data. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 565–572. Springer, Heidelberg (2015). doi:10.1007/978-3-319-24553-9_69

    CrossRef  Google Scholar 

Download references


We would like to thank all anonymous players who participate in our game. We are also grateful to Dr. Mitko Veta for giving us the permission to use the AMIDA13 dataset in our research.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Shadi Albarqouni .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Albarqouni, S., Matl, S., Baust, M., Navab, N., Demirci, S. (2016). Playsourcing: A Novel Concept for Knowledge Creation in Biomedical Research. In: , et al. Deep Learning and Data Labeling for Medical Applications. DLMIA LABELS 2016 2016. Lecture Notes in Computer Science(), vol 10008. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46975-1

  • Online ISBN: 978-3-319-46976-8

  • eBook Packages: Computer ScienceComputer Science (R0)