Skip to main content
Log in

More than data gatherers: exploring player experience in a citizen science game

Quality and User Experience Aims and scope Submit manuscript

Abstract

We present the results of an exploratory player experience study on the game Quantum Moves, a citizen science game where players move quantum particles to help create a quantum computer. Eight-hundred-and-seventeen players responded to a 13-question exploratory survey constructed to understand how players relate to the game, what are their motivations, and how could the game be improved. We analyzed the data using descriptive statistics and thematic analysis. Specifically, the thematic analysis helped identifying two cross-cutting themes amongst the players: (a) learning and (b) the opportunity to contribute to science. Results indicate that the opportunity to help science, along with game design, game elements, involvement of players with the scientific community, and players’ strategies influence experience. Implications of the particular findings for the research on player experience on citizen science games and development of evaluation methods are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. We searched the databases Scopus and Web of Science. Both databases are well-established, multi-disciplinary research platforms, including a wide variety of peer-reviewed journals, and they are being updated regularly. We chose these two databases to ensure relevant papers are included, since it is possible that one database omits relevant research. The Scopus database retrieved 11 articles when searching for TITLE-ABS-KEY (“citizen science” AND “gam*” AND “player experience”). The oldest article dates from 2014. No articles were found when searching (TITLE-ABS-KEY (“citizen science” AND “gam*”)) AND (“quality of experience”). The Web of Science returned one article from 2015 with a similar search string: TOPIC: (“citizen science”) AND TOPIC: (gam*) AND TOPIC: (“player experience*”) and no results when searching TOPIC: (“citizen science”) AND TOPIC: (gam*) AND TOPIC: (“player experience”). Search conducted in 4th September, 2019.

References

  1. Bonney R, Shirk J, Phillips T et al (2014) Next steps for citizen science. Science 343:1436–1437. https://doi.org/10.1126/science.1251554

    Article  Google Scholar 

  2. Tinati R, Luczak-Roesch M, Simperl E, Hall W (2017) An investigation of player motivations in Eyewire, a gamified citizen science project. Comput Hum Behav 73:527–540. https://doi.org/10.1016/j.chb.2016.12.074

    Article  Google Scholar 

  3. Newman G, Wiggins A, Crall A et al (2012) The future of citizen science: emerging technologies and shifting paradigms. Front Ecol Environ 10:298–304. https://doi.org/10.1890/110294

    Article  Google Scholar 

  4. Sørensen JJ, Pedersen MK, Munch M et al (2016) Exploring the quantum speed limit with computer games. Nature 532:210–218

    Article  Google Scholar 

  5. Magnussen R (2017) Involving lay people in research and professional development through gaming: a systematic mapping review. In: Proceedings of the 11th European conference on games based learning, ECGBL 2017, pp 394–401

  6. Weitenberg C, Kuhr S, Mølmer K, Sherson J (2011) A quantum computation architecture using optical tweezers. Phys Rev A. https://doi.org/10.1103/PhysRevA.84.032322

    Article  Google Scholar 

  7. Wiemeyer J, Nacke L, Moser C, Floyd’ Mueller F (2016) Player experience. In: Dörner R, Göbel S, Effelsberg W, Wiemeyer J (eds) Serious games. Foundations, concepts and practice. Springer, Cham

    Google Scholar 

  8. Möller S, Raake A (2014) Quality and quality of experience. Advanced concepts, applications and methods. In: Möller S, Raake A (eds) quality of experience. Springer, Berlin

    Chapter  Google Scholar 

  9. Canossa A, Drachen A (2009) Patterns of play: play-personas in user-centred game development. In: Proceedings of DiGRA 2009. DiGRA

  10. Klimmt C (2003) Dimensions and determinants of the enjoyment of playing digital games: a three-level model. In: Proceedings of level up - 1st international digital games research conference 2003, Netherlands

  11. Csikszentmihalyi M (1990) Flow. The psychology of optimal experience. Harper & Row, Manhattan

    Google Scholar 

  12. Sweetser P, Wyeth P (2005) GameFlow: a model for evaluating player enjoyment in games. Comput Entertain CIE Theor Pract Comput Appl Entertain 3:3. https://doi.org/10.1145/1077246.1077253

    Article  Google Scholar 

  13. Brown E, Cairns P (2004) A grounded investigation of game immersion. In: CHI’04 extended abstracts on human factors in computing systems. ACM, Austria, pp 24–29

  14. Klimmt C, Hartmann T, Frey A (2007) Effectance and control as determinants of video game enjoyment. Cyberpsychol Behav 10:845–848. https://doi.org/10.1089/cpb.2007.9942

    Article  Google Scholar 

  15. Poels K, de Kort Y, Ijsselsteijn W (2012) Identification and categorization of digital game experiences: a qualitative study integrating theoretical insights and player perspectives. Westminst Pap 9:107–129. https://doi.org/10.16997/wpcc.153

    Article  Google Scholar 

  16. Le Callet P, Möller S, Perkis A (series ed) (2013) Qualinet white paper on definitions of quality of experience (QoE) and related concepts [white paper]. Retrieved from http://www.qualinet.eu/

  17. Nacke L, Drachen A, Göbel S (2010) Methods for evaluating gameplay experience in a serious gaming context. Int J Comput Sci Sport 9:40–51

    Google Scholar 

  18. Law E, von Ahn L (2011) Human computation. Morgan & Claypool, San Rafael

    Book  Google Scholar 

  19. Deterding S, Sicart M, Nacke L, et al (2011) Gamification: using game design elements in non-gaming contexts. In: CHI’11 extended abstracts on human factors in computing systems. ACM Press, Toronto, pp 2425–2428

  20. Ponti M, Hillman T, Kullenberg C, Kasperowski D (2018) Getting it right or being top rank: games in citizen science. Citiz Sci Theory Pract 3:1–12. https://doi.org/10.5334/cstp.101

    Article  Google Scholar 

  21. Bowser A, Hansen D, He Y, et al (2013) Using gamification to inspire new citizen science volunteers. In: Proceedings of the companion publication of the 17th ACM conference on computer supported cooperative work and social computing. ACM Press, Toronto, pp 18–25

  22. Curtis V (2016) Motivation to participate in an online citizen science game: a study of Foldit. Sci Commun 37:723–746. https://doi.org/10.1177/1075547015609322

    Article  Google Scholar 

  23. Greenhill A, Holmes K, Woodcock J et al (2016) Playing with science: exploring how game activity motivates users participation on an online citizen science platform. Aslib J Inf Manag 68:306–325. https://doi.org/10.1108/AJIM-11-2015-0182

    Article  Google Scholar 

  24. Iacovides I, Jennet C, Cornish-Trestrail C, Cox A (2013) Do games attract or sustain engagement in citizen science? A study of volunteer motivations. In: CHI’13 extended abstracts on human factors in computing systems. ACM Press, Paris, pp 1101–1106

  25. Cox A, Cairns P, Shah P, Carroll M (2012) Not doing but thinking: the role of challenge in the gaming experience. In: Proceedings of CHI’12 SIGCHI conference on human factors in computing systems. ACM, Texas, pp 79–88

  26. Prestopnik N, Tang J (2015) Points, stories, worlds, and diegesis: comparing player experiences in two citizen science games. Comput Hum Behav 52:492–506. https://doi.org/10.1016/j.chb.2015.05.051

    Article  Google Scholar 

  27. Hess M, Wiemeyer J, Hamacher K, Goesele M (2014) Serious games for solving protein sequence alignments—combining citizen science and gaming. In: Göbel S, Wiemeyer J (eds) Games for training, education, health and sports. Springer, Berlin

    Google Scholar 

  28. Pedersen MK, Rasmussen N, Sherson J, Basaiawmoit R (2017) Leaderboard effects on player performance in a citizen science game. In: Proceedings of the 11th European conference on games based learning

  29. Lieberoth A, Pedersen MK, Marin A et al (2014) Getting humans to do quantum optimization: user acquisition, engagement and early results from the citizen cyberscience game quantum moves. Hum Comput 1:219–244. https://doi.org/10.15346/hc.v1i2.11

    Article  Google Scholar 

  30. Directorate-General for Research Science, Economy and Society (2010) European textbook on ethics in research. Publication office of the European Union, Luxembourg

    Google Scholar 

  31. González Mariño JC, Cantú Gallegos ML, Camacho-Cruz HE (2018) Redesigning the Bartle test of gamer psychology for its application in gamification processes of learning. In: Proceedings of the 12th international multi-conference on society, cybernetics and informatics. IIIS, Florida, pp 35–40

  32. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3:77–101. https://doi.org/10.1191/1478088706qp063oa

    Article  Google Scholar 

  33. Campbell J, Quincy C, Osserman J, Pedersen O (2013) Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociol Methods Res 42:294–320. https://doi.org/10.1177/0049124113500475smr.sagepub.com

    Article  MathSciNet  Google Scholar 

  34. Eccleston P, Werneke U, Armon K et al (2001) Accounting for overlap? An application of Mezzich’s kappa statistic to test interrater reliability of interview data on parental accident and emergency attendance. J Adv Nurs 33:784–790. https://doi.org/10.1046/j.1365-2648.2001.01718.x

    Article  Google Scholar 

  35. Stokke A (2015) What to do about Canada’s declining math scores. C. D. Howe Institute Commentary. https://doi.org/10.2139/ssrn.2613146

  36. Dean D, Kuhn D (2006) Direct instruction vs. discovery: the long view. Science Education 91(3):384–397. https://doi.org/10.1002/sce.20194

    Article  Google Scholar 

  37. Tinati R, van Kleek M, Simperl E, et al (2015) Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, Seoul, pp 4069–4078

  38. Habgood J, Overmars M (2006) The game maker’s apprentice. Game development for beginners. Apress, Berkeley

    Book  Google Scholar 

  39. Seaborn K, Fels D (2014) Gamification in theory and action: a survey. Int J Hum-Comput Stud 74:14–31. https://doi.org/10.1016/j.ijhcs.2014.09.006

    Article  Google Scholar 

  40. Franzoni C, Sauermann H (2014) Crowd science: the organization of scientific research in open collaborative projects. Res Policy 43:1–20. https://doi.org/10.1016/j.respol.2013.07.005

    Article  Google Scholar 

  41. Elias G, Garfield R, Gutschera R et al (2012) Characteristics of games. MIT Press, Cambridge

    Google Scholar 

  42. Azwihangwisi M (2015) Piaget’s theory of human development and education. Int Encycl Soc Behav Sci 18:125–132

    Google Scholar 

  43. Horstman T, Chen M (2012) Gamers as scientists? The relationship between participating in foldit play and doing science. In: Presented at the American educational research association (AERA), Canada. Retrieved from https://www.researchgate.net/publication/258294401_Gamers_as_Scientists_The_Relationship_Between_Participating_in_Foldit_Play_and_Doing_Science

  44. Jennett C, Kloetzer L, Schneider D et al (2016) Motivations, learning and creativity in online citizen science. J Sci Commun 15:1–23

    Article  Google Scholar 

  45. Lafourcade M, Joubert A, Le Brun N (2015) Games with a purpose (GWAPs). Wiley, New York

    Book  Google Scholar 

  46. Prestopnik N, Crowston K, Wang J (2017) Gamers, citizen scientists, and data: exploring participant contributions in two games with a purpose. Comput Hum Behav 68:254–268. https://doi.org/10.1016/j.chb.2016.11.035

    Article  Google Scholar 

  47. Skarlatidou A, Ponti M, Sprinks J et al (2019) User experience of digital technologies in citizen science. J Sci Commun. https://doi.org/10.22323/2.18010501

    Article  Google Scholar 

  48. Brockmyer J, Fox C, Curtiss K et al (2009) The development of the game engagement questionnaire: a measure of engagement in video game-playing. J Exp Soc Psychol 45:624–634. https://doi.org/10.1016/j.jesp.2009.02.016

    Article  Google Scholar 

  49. Johnson D, Gardner J, Perry R (2018) Validation of two game experience scales: the player experience of need satisfaction (PENS) and game experience questionnaire (GEQ). Int J Hum-Comput Stud 118:38–46. https://doi.org/10.1016/j.ijhcs.2018.05.003

    Article  Google Scholar 

  50. Preece J (2016) Citizen science: new research challenges for human–computer interaction. Int J Hum-Comput Interact 32:585–612. https://doi.org/10.1080/10447318.2016.1194153

    Article  Google Scholar 

  51. Kim S, Robson C, Zimmerman T, et al (2011) Creek watch: pairing usefulness and usability for successful citizen science. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’11). ACM Press, New York, pp 2125–2134

Download references

Acknowledgements

This work was supported through funding from Marianne and Marcus Wallenberg Grant No. 2013.0020, Lundbeck Foundation, Grant No. R139-2012-12633, Carlsberg Foundation Grant No. CF18-0019, and John Templeton Foundation Grant No. 60969. J.F.S. Acknowledges funding from the ERC, H2020 Grant No. 639560 (MECTRL) and the Synakos foundation.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Carlos Díaz.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

Distribution of players according to the player types as assessed by the Bartle Test (Q20). NA stands for the players who could not take the test. The total sum of players per category is more than the players who answer the question; this is because several players reported that they belong to more than one category.

figure a

Appendix 2

Coding schema of player responses to question Q19

Topics and subtopics (indented)

Definition

Not defined—no response

Players did not answer the question, or their answers were irrelevant

Game type

Players use a name that defines the game as a genre and explain why they like it or not

 Casual game

Players state the game is a casual game or they play it casually

 Zen game

Players state the game is a Zen game or having calming properties

 Puzzle

Players state the game is a puzzle

 Single player game

Players state the game is a single-player game

Distinct game

Players state the game is different from other games on the market and/or other citizen science games

Educational effects

Players state the game has educational effects, or it is good for learning/teaching science.

 Interest in learning

Players state the game taught them something about science

 Generating curiosity

Players state the game generated curiosity for them to learn more

Engagement

Assigned to indicate the way players engaged with the game

 Enjoyment

Players state they have enjoyed the game

 Monotony

Players state the game is monotonous or repetitive

 Boredom

Players state the game is boring

 Frustration

Players state the game is frustrating, either for its mechanics or for problems with the game build

 Challenge

Players state the game is challenging, either in a positive or a negative way

 Confidence

Players state they are confident when solving the problems proposed by the game

Modes of learning

Applied when players mention how the game engaged them in learning, or they suggest a way in which they can learn about the game or from the game

 Vicarious learning

Players would like to see other players’ solutions to learn better ways to solve problems

 Interest in quantum mechanics

Players state they have learned more about the specific field of science of the game

Game design suggestions

Applied when players suggest design ideas for improving the game

 Graphic design suggestions

Players suggest changes to the graphic aspects of the game

 Opportunities for player customization

Players state they would like to be able to change certain features of the game ad libitum (e.g., re-skin the game)

 Send reminders

Players state they would like to receive periodic reminders to play the game

 Sound design suggestions

Players suggest sounds designs for the game, such as auditory cues for success or background music

 New mechanics suggestions

Players suggest new or improved mechanics for the game

 Enlarge game

Players suggest developing the game further, for example by adding levels

Improving tutorials

Players state tutorials should be improved, or they evaluated the current tutorial as insufficient

Influence of leader board on motivation

Players evaluate how the leaderboard influenced their motivation when playing the game

Interest in contributing to citizen science

Players state their interest in the game for their contribution to citizen science

Intuitive (easy to understand)

Players state the game and game mechanics were easy to understand, or they did not have to make mental effort to understand it and play it

Intuitive solution

Players state the game was easy to play and not much knowledge was required to solve the problems

Knowledge representation

Players state the way the game represents information on quantum physics makes it easy to understand this topic

Lack of game feedback

Players state not having enough feedback from the game

Lack of information regarding quantum physics (scientific topic)

Players state the game lacked of more and deeper information about quantum physics

Lack of interaction with the scientific team and the game developers

Players state the need for more interaction and conversations with the team of scientist and developers behind the game to better understand how the data gathered is being used

Lack of understanding

Applied when players state not being able to understand certain aspects of the game

 Lack of understanding of the game

Players state they did not understand the game

 Lack of understanding of the game mechanics

Players state they did not understand the game mechanics

 Lack of understanding of the procedural understanding

Players state they did not understand the procedures required to solve the problems

 Lack of understanding of the UI interface

Players state they did not understand the user interface or they were confused by it

 Lack of understanding of the purpose of the game

Players state they did not understand the purpose of the game apart from the partial goals in each level

Learning curve

Players make a statement on the learning curve of the game. Statements about game progression, level progression, and blockages in the gameplay fall into this category

Parallel thinking

Participant state having to think in non-conventional ways to solve the problems proposed in the game

Player types

Players state they engage or not engage with the game due to a personal trait

 Competitive

Players state they are competitive or like competition

 Non-competitive

Players state they are not competitive or dislike competition

 Not a gamer

Players state they are regular players of video games

 Puzzler

Players state they like solving puzzles

Psychophysical constraints

Players state having problems performing tasks requiring fine motor skills

Relationship between game and the scientific topic (QP)

Players state that it would be a gain for the game if it highlighted more its scientific relevance

Relevance of controllers

Players state controllers were relevant for the implementation of the strategy, usually in a negative way

Relevance of gaming experience

Player state that their experience playing games helped them solve the challenges proposed by the game

Relevance of science knowledge

Player state their knowledge of science helped them solve the challenges proposed by the game

Rewards other than the score system

Players state they find rewarding aspects of the game that are different from the score system, e.g., learning science or helping the development of a scientific tool

Science dissemination

Players state the game is good for science dissemination

Social aspects

Players state interest in the inclusion of social aspects in the game

Strategy making

Applied when players talked about the strategies they used to solve the challenges proposed by the game

 Lack of strategy

Players state not having a strategy

 Scientific reasoning/experimentation

Players state experimenting and finding patterns which allow them to overcome the game challenges

 Serendipity

Players solved the problem by chance or luck

 Trial and error

Players solved the game challenges using trial and error

Technical issues

Applied when players state having technical issues with the game

 Problems with responsiveness and accuracy

Players state having problems with the responsiveness or the accuracy of the game depending on the device they used

 Game bugs

Players reported bugs in the game

 Relevance of OS

Players reported incompatibility of the game with some operative systems

 Relevance of console

Players state the device (e.g., PC or mobile device) they used for playing influenced their performance

 Problems with data transmission

Players reported to stop playing because data transmission was very costly

Too busy to play

Players state not playing much because they had other activities to do

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Díaz, C., Ponti, M., Haikka, P. et al. More than data gatherers: exploring player experience in a citizen science game. Qual User Exp 5, 1 (2020). https://doi.org/10.1007/s41233-019-0030-8

Download citation

  • Received:

  • Published:

  • DOI: https://doi.org/10.1007/s41233-019-0030-8

Keywords

Navigation