Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

More than data gatherers: exploring player experience in a citizen science game

Abstract

We present the results of an exploratory player experience study on the game Quantum Moves, a citizen science game where players move quantum particles to help create a quantum computer. Eight-hundred-and-seventeen players responded to a 13-question exploratory survey constructed to understand how players relate to the game, what are their motivations, and how could the game be improved. We analyzed the data using descriptive statistics and thematic analysis. Specifically, the thematic analysis helped identifying two cross-cutting themes amongst the players: (a) learning and (b) the opportunity to contribute to science. Results indicate that the opportunity to help science, along with game design, game elements, involvement of players with the scientific community, and players’ strategies influence experience. Implications of the particular findings for the research on player experience on citizen science games and development of evaluation methods are discussed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    We searched the databases Scopus and Web of Science. Both databases are well-established, multi-disciplinary research platforms, including a wide variety of peer-reviewed journals, and they are being updated regularly. We chose these two databases to ensure relevant papers are included, since it is possible that one database omits relevant research. The Scopus database retrieved 11 articles when searching for TITLE-ABS-KEY (“citizen science” AND “gam*” AND “player experience”). The oldest article dates from 2014. No articles were found when searching (TITLE-ABS-KEY (“citizen science” AND “gam*”)) AND (“quality of experience”). The Web of Science returned one article from 2015 with a similar search string: TOPIC: (“citizen science”) AND TOPIC: (gam*) AND TOPIC: (“player experience*”) and no results when searching TOPIC: (“citizen science”) AND TOPIC: (gam*) AND TOPIC: (“player experience”). Search conducted in 4th September, 2019.

References

  1. 1.

    Bonney R, Shirk J, Phillips T et al (2014) Next steps for citizen science. Science 343:1436–1437. https://doi.org/10.1126/science.1251554

  2. 2.

    Tinati R, Luczak-Roesch M, Simperl E, Hall W (2017) An investigation of player motivations in Eyewire, a gamified citizen science project. Comput Hum Behav 73:527–540. https://doi.org/10.1016/j.chb.2016.12.074

  3. 3.

    Newman G, Wiggins A, Crall A et al (2012) The future of citizen science: emerging technologies and shifting paradigms. Front Ecol Environ 10:298–304. https://doi.org/10.1890/110294

  4. 4.

    Sørensen JJ, Pedersen MK, Munch M et al (2016) Exploring the quantum speed limit with computer games. Nature 532:210–218

  5. 5.

    Magnussen R (2017) Involving lay people in research and professional development through gaming: a systematic mapping review. In: Proceedings of the 11th European conference on games based learning, ECGBL 2017, pp 394–401

  6. 6.

    Weitenberg C, Kuhr S, Mølmer K, Sherson J (2011) A quantum computation architecture using optical tweezers. Phys Rev A. https://doi.org/10.1103/PhysRevA.84.032322

  7. 7.

    Wiemeyer J, Nacke L, Moser C, Floyd’ Mueller F (2016) Player experience. In: Dörner R, Göbel S, Effelsberg W, Wiemeyer J (eds) Serious games. Foundations, concepts and practice. Springer, Cham

  8. 8.

    Möller S, Raake A (2014) Quality and quality of experience. Advanced concepts, applications and methods. In: Möller S, Raake A (eds) quality of experience. Springer, Berlin

  9. 9.

    Canossa A, Drachen A (2009) Patterns of play: play-personas in user-centred game development. In: Proceedings of DiGRA 2009. DiGRA

  10. 10.

    Klimmt C (2003) Dimensions and determinants of the enjoyment of playing digital games: a three-level model. In: Proceedings of level up - 1st international digital games research conference 2003, Netherlands

  11. 11.

    Csikszentmihalyi M (1990) Flow. The psychology of optimal experience. Harper & Row, Manhattan

  12. 12.

    Sweetser P, Wyeth P (2005) GameFlow: a model for evaluating player enjoyment in games. Comput Entertain CIE Theor Pract Comput Appl Entertain 3:3. https://doi.org/10.1145/1077246.1077253

  13. 13.

    Brown E, Cairns P (2004) A grounded investigation of game immersion. In: CHI’04 extended abstracts on human factors in computing systems. ACM, Austria, pp 24–29

  14. 14.

    Klimmt C, Hartmann T, Frey A (2007) Effectance and control as determinants of video game enjoyment. Cyberpsychol Behav 10:845–848. https://doi.org/10.1089/cpb.2007.9942

  15. 15.

    Poels K, de Kort Y, Ijsselsteijn W (2012) Identification and categorization of digital game experiences: a qualitative study integrating theoretical insights and player perspectives. Westminst Pap 9:107–129. https://doi.org/10.16997/wpcc.153

  16. 16.

    Le Callet P, Möller S, Perkis A (series ed) (2013) Qualinet white paper on definitions of quality of experience (QoE) and related concepts [white paper]. Retrieved from http://www.qualinet.eu/

  17. 17.

    Nacke L, Drachen A, Göbel S (2010) Methods for evaluating gameplay experience in a serious gaming context. Int J Comput Sci Sport 9:40–51

  18. 18.

    Law E, von Ahn L (2011) Human computation. Morgan & Claypool, San Rafael

  19. 19.

    Deterding S, Sicart M, Nacke L, et al (2011) Gamification: using game design elements in non-gaming contexts. In: CHI’11 extended abstracts on human factors in computing systems. ACM Press, Toronto, pp 2425–2428

  20. 20.

    Ponti M, Hillman T, Kullenberg C, Kasperowski D (2018) Getting it right or being top rank: games in citizen science. Citiz Sci Theory Pract 3:1–12. https://doi.org/10.5334/cstp.101

  21. 21.

    Bowser A, Hansen D, He Y, et al (2013) Using gamification to inspire new citizen science volunteers. In: Proceedings of the companion publication of the 17th ACM conference on computer supported cooperative work and social computing. ACM Press, Toronto, pp 18–25

  22. 22.

    Curtis V (2016) Motivation to participate in an online citizen science game: a study of Foldit. Sci Commun 37:723–746. https://doi.org/10.1177/1075547015609322

  23. 23.

    Greenhill A, Holmes K, Woodcock J et al (2016) Playing with science: exploring how game activity motivates users participation on an online citizen science platform. Aslib J Inf Manag 68:306–325. https://doi.org/10.1108/AJIM-11-2015-0182

  24. 24.

    Iacovides I, Jennet C, Cornish-Trestrail C, Cox A (2013) Do games attract or sustain engagement in citizen science? A study of volunteer motivations. In: CHI’13 extended abstracts on human factors in computing systems. ACM Press, Paris, pp 1101–1106

  25. 25.

    Cox A, Cairns P, Shah P, Carroll M (2012) Not doing but thinking: the role of challenge in the gaming experience. In: Proceedings of CHI’12 SIGCHI conference on human factors in computing systems. ACM, Texas, pp 79–88

  26. 26.

    Prestopnik N, Tang J (2015) Points, stories, worlds, and diegesis: comparing player experiences in two citizen science games. Comput Hum Behav 52:492–506. https://doi.org/10.1016/j.chb.2015.05.051

  27. 27.

    Hess M, Wiemeyer J, Hamacher K, Goesele M (2014) Serious games for solving protein sequence alignments—combining citizen science and gaming. In: Göbel S, Wiemeyer J (eds) Games for training, education, health and sports. Springer, Berlin

  28. 28.

    Pedersen MK, Rasmussen N, Sherson J, Basaiawmoit R (2017) Leaderboard effects on player performance in a citizen science game. In: Proceedings of the 11th European conference on games based learning

  29. 29.

    Lieberoth A, Pedersen MK, Marin A et al (2014) Getting humans to do quantum optimization: user acquisition, engagement and early results from the citizen cyberscience game quantum moves. Hum Comput 1:219–244. https://doi.org/10.15346/hc.v1i2.11

  30. 30.

    Directorate-General for Research Science, Economy and Society (2010) European textbook on ethics in research. Publication office of the European Union, Luxembourg

  31. 31.

    González Mariño JC, Cantú Gallegos ML, Camacho-Cruz HE (2018) Redesigning the Bartle test of gamer psychology for its application in gamification processes of learning. In: Proceedings of the 12th international multi-conference on society, cybernetics and informatics. IIIS, Florida, pp 35–40

  32. 32.

    Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3:77–101. https://doi.org/10.1191/1478088706qp063oa

  33. 33.

    Campbell J, Quincy C, Osserman J, Pedersen O (2013) Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociol Methods Res 42:294–320. https://doi.org/10.1177/0049124113500475smr.sagepub.com

  34. 34.

    Eccleston P, Werneke U, Armon K et al (2001) Accounting for overlap? An application of Mezzich’s kappa statistic to test interrater reliability of interview data on parental accident and emergency attendance. J Adv Nurs 33:784–790. https://doi.org/10.1046/j.1365-2648.2001.01718.x

  35. 35.

    Stokke A (2015) What to do about Canada’s declining math scores. C. D. Howe Institute Commentary. https://doi.org/10.2139/ssrn.2613146

  36. 36.

    Dean D, Kuhn D (2006) Direct instruction vs. discovery: the long view. Science Education 91(3):384–397. https://doi.org/10.1002/sce.20194

  37. 37.

    Tinati R, van Kleek M, Simperl E, et al (2015) Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, Seoul, pp 4069–4078

  38. 38.

    Habgood J, Overmars M (2006) The game maker’s apprentice. Game development for beginners. Apress, Berkeley

  39. 39.

    Seaborn K, Fels D (2014) Gamification in theory and action: a survey. Int J Hum-Comput Stud 74:14–31. https://doi.org/10.1016/j.ijhcs.2014.09.006

  40. 40.

    Franzoni C, Sauermann H (2014) Crowd science: the organization of scientific research in open collaborative projects. Res Policy 43:1–20. https://doi.org/10.1016/j.respol.2013.07.005

  41. 41.

    Elias G, Garfield R, Gutschera R et al (2012) Characteristics of games. MIT Press, Cambridge

  42. 42.

    Azwihangwisi M (2015) Piaget’s theory of human development and education. Int Encycl Soc Behav Sci 18:125–132

  43. 43.

    Horstman T, Chen M (2012) Gamers as scientists? The relationship between participating in foldit play and doing science. In: Presented at the American educational research association (AERA), Canada. Retrieved from https://www.researchgate.net/publication/258294401_Gamers_as_Scientists_The_Relationship_Between_Participating_in_Foldit_Play_and_Doing_Science

  44. 44.

    Jennett C, Kloetzer L, Schneider D et al (2016) Motivations, learning and creativity in online citizen science. J Sci Commun 15:1–23

  45. 45.

    Lafourcade M, Joubert A, Le Brun N (2015) Games with a purpose (GWAPs). Wiley, New York

  46. 46.

    Prestopnik N, Crowston K, Wang J (2017) Gamers, citizen scientists, and data: exploring participant contributions in two games with a purpose. Comput Hum Behav 68:254–268. https://doi.org/10.1016/j.chb.2016.11.035

  47. 47.

    Skarlatidou A, Ponti M, Sprinks J et al (2019) User experience of digital technologies in citizen science. J Sci Commun. https://doi.org/10.22323/2.18010501

  48. 48.

    Brockmyer J, Fox C, Curtiss K et al (2009) The development of the game engagement questionnaire: a measure of engagement in video game-playing. J Exp Soc Psychol 45:624–634. https://doi.org/10.1016/j.jesp.2009.02.016

  49. 49.

    Johnson D, Gardner J, Perry R (2018) Validation of two game experience scales: the player experience of need satisfaction (PENS) and game experience questionnaire (GEQ). Int J Hum-Comput Stud 118:38–46. https://doi.org/10.1016/j.ijhcs.2018.05.003

  50. 50.

    Preece J (2016) Citizen science: new research challenges for human–computer interaction. Int J Hum-Comput Interact 32:585–612. https://doi.org/10.1080/10447318.2016.1194153

  51. 51.

    Kim S, Robson C, Zimmerman T, et al (2011) Creek watch: pairing usefulness and usability for successful citizen science. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’11). ACM Press, New York, pp 2125–2134

Download references

Acknowledgements

This work was supported through funding from Marianne and Marcus Wallenberg Grant No. 2013.0020, Lundbeck Foundation, Grant No. R139-2012-12633, Carlsberg Foundation Grant No. CF18-0019, and John Templeton Foundation Grant No. 60969. J.F.S. Acknowledges funding from the ERC, H2020 Grant No. 639560 (MECTRL) and the Synakos foundation.

Author information

Correspondence to Carlos Díaz.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

Distribution of players according to the player types as assessed by the Bartle Test (Q20). NA stands for the players who could not take the test. The total sum of players per category is more than the players who answer the question; this is because several players reported that they belong to more than one category.

figurea

Appendix 2

Coding schema of player responses to question Q19

Topics and subtopics (indented) Definition
Not defined—no response Players did not answer the question, or their answers were irrelevant
Game type Players use a name that defines the game as a genre and explain why they like it or not
 Casual game Players state the game is a casual game or they play it casually
 Zen game Players state the game is a Zen game or having calming properties
 Puzzle Players state the game is a puzzle
 Single player game Players state the game is a single-player game
Distinct game Players state the game is different from other games on the market and/or other citizen science games
Educational effects Players state the game has educational effects, or it is good for learning/teaching science.
 Interest in learning Players state the game taught them something about science
 Generating curiosity Players state the game generated curiosity for them to learn more
Engagement Assigned to indicate the way players engaged with the game
 Enjoyment Players state they have enjoyed the game
 Monotony Players state the game is monotonous or repetitive
 Boredom Players state the game is boring
 Frustration Players state the game is frustrating, either for its mechanics or for problems with the game build
 Challenge Players state the game is challenging, either in a positive or a negative way
 Confidence Players state they are confident when solving the problems proposed by the game
Modes of learning Applied when players mention how the game engaged them in learning, or they suggest a way in which they can learn about the game or from the game
 Vicarious learning Players would like to see other players’ solutions to learn better ways to solve problems
 Interest in quantum mechanics Players state they have learned more about the specific field of science of the game
Game design suggestions Applied when players suggest design ideas for improving the game
 Graphic design suggestions Players suggest changes to the graphic aspects of the game
 Opportunities for player customization Players state they would like to be able to change certain features of the game ad libitum (e.g., re-skin the game)
 Send reminders Players state they would like to receive periodic reminders to play the game
 Sound design suggestions Players suggest sounds designs for the game, such as auditory cues for success or background music
 New mechanics suggestions Players suggest new or improved mechanics for the game
 Enlarge game Players suggest developing the game further, for example by adding levels
Improving tutorials Players state tutorials should be improved, or they evaluated the current tutorial as insufficient
Influence of leader board on motivation Players evaluate how the leaderboard influenced their motivation when playing the game
Interest in contributing to citizen science Players state their interest in the game for their contribution to citizen science
Intuitive (easy to understand) Players state the game and game mechanics were easy to understand, or they did not have to make mental effort to understand it and play it
Intuitive solution Players state the game was easy to play and not much knowledge was required to solve the problems
Knowledge representation Players state the way the game represents information on quantum physics makes it easy to understand this topic
Lack of game feedback Players state not having enough feedback from the game
Lack of information regarding quantum physics (scientific topic) Players state the game lacked of more and deeper information about quantum physics
Lack of interaction with the scientific team and the game developers Players state the need for more interaction and conversations with the team of scientist and developers behind the game to better understand how the data gathered is being used
Lack of understanding Applied when players state not being able to understand certain aspects of the game
 Lack of understanding of the game Players state they did not understand the game
 Lack of understanding of the game mechanics Players state they did not understand the game mechanics
 Lack of understanding of the procedural understanding Players state they did not understand the procedures required to solve the problems
 Lack of understanding of the UI interface Players state they did not understand the user interface or they were confused by it
 Lack of understanding of the purpose of the game Players state they did not understand the purpose of the game apart from the partial goals in each level
Learning curve Players make a statement on the learning curve of the game. Statements about game progression, level progression, and blockages in the gameplay fall into this category
Parallel thinking Participant state having to think in non-conventional ways to solve the problems proposed in the game
Player types Players state they engage or not engage with the game due to a personal trait
 Competitive Players state they are competitive or like competition
 Non-competitive Players state they are not competitive or dislike competition
 Not a gamer Players state they are regular players of video games
 Puzzler Players state they like solving puzzles
Psychophysical constraints Players state having problems performing tasks requiring fine motor skills
Relationship between game and the scientific topic (QP) Players state that it would be a gain for the game if it highlighted more its scientific relevance
Relevance of controllers Players state controllers were relevant for the implementation of the strategy, usually in a negative way
Relevance of gaming experience Player state that their experience playing games helped them solve the challenges proposed by the game
Relevance of science knowledge Player state their knowledge of science helped them solve the challenges proposed by the game
Rewards other than the score system Players state they find rewarding aspects of the game that are different from the score system, e.g., learning science or helping the development of a scientific tool
Science dissemination Players state the game is good for science dissemination
Social aspects Players state interest in the inclusion of social aspects in the game
Strategy making Applied when players talked about the strategies they used to solve the challenges proposed by the game
 Lack of strategy Players state not having a strategy
 Scientific reasoning/experimentation Players state experimenting and finding patterns which allow them to overcome the game challenges
 Serendipity Players solved the problem by chance or luck
 Trial and error Players solved the game challenges using trial and error
Technical issues Applied when players state having technical issues with the game
 Problems with responsiveness and accuracy Players state having problems with the responsiveness or the accuracy of the game depending on the device they used
 Game bugs Players reported bugs in the game
 Relevance of OS Players reported incompatibility of the game with some operative systems
 Relevance of console Players state the device (e.g., PC or mobile device) they used for playing influenced their performance
 Problems with data transmission Players reported to stop playing because data transmission was very costly
Too busy to play Players state not playing much because they had other activities to do

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Díaz, C., Ponti, M., Haikka, P. et al. More than data gatherers: exploring player experience in a citizen science game. Qual User Exp 5, 1 (2020). https://doi.org/10.1007/s41233-019-0030-8

Download citation

Keywords

  • Citizen science
  • Games with a purpose
  • Player experience
  • Quantum physics