User Modeling and User-Adapted Interaction

, Volume 24, Issue 5, pp 413–451

A trace-based approach to identifying users’ engagement and qualifying their engaged-behaviours in interactive systems: application to a social game

Article

Abstract

Analysing and monitoring users’ engaged-behaviours continuously and under ecologically valid conditions can reveal valuable information for designers and practitioners, allowing them to analyse, design and monitor the interactive mediated activity, and then to adapt and personalise it. An interactive mediated activity is a human activity supported by digital interactive technologies. While classical metric methods fall within quantitative approaches, this paper proposes a qualitative approach to identifying users’ engagement and qualifying their engaged-behaviours from their traces of interaction. Traces of interaction represent the users’ activities with an interactive environment. The basis of our approach is to transform low-level traces of interaction into meaningful information represented in higher-level traces. For this, our approach combines three theoretical frameworks: the Self-Determination Theory, the Activity Theory and the Trace Theory. Our approach has been implemented and tested in the context of the QUEJANT Projet. QUEJANT targets the development of a system allowing the actors of Social Gaming to analyse players’ engagement from an analysis of their activity traces. In order to demonstrate the feasibility of our approach, we implemented the whole process in a prototype and applied it to 12 players’ interaction data collected over four months. Based on these interaction data, we were able to identify engaged and non-engaged users and to qualify their types of engaged-behaviours. We also conducted a user study based on a validation of our results by experts. The high prediction rate obtained confirms the performance of our approach. We finally discuss the limitations of our approach, the potential fields of application and the implications for digital behavioural interventions.

Keywords

Engagement assessment Engaged behaviours Qualitative approach User behaviour analysis Self-Determination Theory Activity Theory  Interaction traces Social game 

References

  1. Andersen, E., O’Rourke, E., Liu, Y.E., Snider, R., Lowdermilk, J., Truong, D., Cooper, S., Popovic, Z.: The impact of tutorials on games of varying complexity. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 59–68. ACM, Austin, Texas, USA, CHI 2012 (2012)Google Scholar
  2. Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. In: Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling, pp. 17–24. IOS Press, Brighton, UK (2009)Google Scholar
  3. Attfield, S., Kazai, G., Lalmas, M., Piwowarski, B.: Towards a science of user engagement (position paper). In: WSDM Workshop on User Modelling for Web Applications, Hong Kong, China (2011)Google Scholar
  4. Barak, A., Hen, L., Boniel-Nissim, M., Shapira, N.: A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. J. Technol. Human Serv. 26(2–4), 109–160 (2008)CrossRefGoogle Scholar
  5. Baranowski, T., Baranowski, J., Cullen, K.W., Marsh, T., Islam, N., Zakeri, I., Honess-Morreale, L., deMoor, C.: Squire’s quest! Dietary outcome evaluation of a multimedia game. Am. J. Prev. Med. 24(1), 52–61 (2003)CrossRefGoogle Scholar
  6. Baranowski, T., Buday, R., Thompson, D.I., Baranowski, J.: Playing for real: video games and stories for health-related behavior change. Am. J. Prev. Med. 34(1), 74–82 (2008)CrossRefGoogle Scholar
  7. Bardram, J.,E.: Plans as situated action: an activity theory approach to workflow systems. In: Proceedings of the Fifth Conference on European Conference on Computer-Supported Cooperative Work, ECSCW’97, pp. 17–32. Kluwer, Lancaster, UK (1997)Google Scholar
  8. Bauckhage, C., Kersting, K., Sifa, R., Thurau, C., Drachen, A., Canossa, A.: How players lose interest in playing a game: an empirical study based on distributions of total playing times. In: Lucas, S., Cho, S.-B., M.S.E. (eds.) Proceedings of the IEEE Conference on Computational Intelligence and Games, CIG 2012 , pp. 139–146. Granada, Spain (2012)Google Scholar
  9. Beal, C.R., Qu, L., Lee, H.: Classifying learner engagement through integration of multiple data sources. In: Proceedings of the 21st National Conference on Artificial Intelligence—Volume 1, AAAI’06. Boston, Massachusetts, pp. 151–156 (2006)Google Scholar
  10. Bianchi-Berthouze, N.: Understanding the role of body movement in player engagement. Human Comput. Interact. 28(1), 40–75 (2013)Google Scholar
  11. Bouvier, P., Lavoué, E., Sehaba, K., George, S.: Identifying learner’s engagement in learning games—a qualitative approach based on learner’s traces of interaction. In: Foley, O., Restivo, M.T., Uhomoibhi, J.O., Helfert, M. (eds.) Proceedings of the 5th International Conference on Computer Supported Education, CSEDU 2013, pp. 339–350. SciTePress, Aachen, Germany (2013a)Google Scholar
  12. Bouvier, P., Sehaba, K., Lavoué, E., George, S.: Using traces to qualify learner’s engagement in game-based learning. In: Proceedings of The 13th IEEE International Conference on Advanced Learning Technologies, ICALT 2013, pp. 432–436. IEEE Computer Society, Beijing, China (2013b)Google Scholar
  13. Boyle, E.A., Connolly, T.M., Hainey, T., Boyle, J.M.: Engagement in digital entertainment games: a systematic review. Comput. Human Behav. 28(3), 771–780 (2012)CrossRefGoogle Scholar
  14. Brockmyer, J.H., Fox, C.M., Curtiss, K.A., McBroom, E., Burkhart, K.M., Pidruzny, J.N.: The development of the game engagement questionnaire: a measure of engagement in video game-playing. J. Exp. Soc. Psychol. 45(4), 624–634 (2009)Google Scholar
  15. Brown, E., Cairns, P.:A grounded investigation of game immersion. In: CHI ’04 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2004, pp. 1297–1300. Vienna, Austria (2004)Google Scholar
  16. Buder, J., Bodemer, D.:Group awareness tools for controversial cscl discussions: dissociating rating effects and visualized feedback effects. In: 9th International Conference on Computer Supported Collaborative Learning, CSCL 2011, pp. 358–365, Hong Kong (2011)Google Scholar
  17. Calvo, R., D’Mello, S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing 1(1), 18–37 (2010)CrossRefGoogle Scholar
  18. Camerini, L., Giacobazzi, M., Boneschi, M., Schulz, P.J., Rubinelli, S.: Design and implementation of a web-based tailored gymnasium to enhance self-management of fibromyalgia. User Model User-Adapt Interact 21(4–5), 485–511 (2011)Google Scholar
  19. Canossa, A., Drachen, A.: Patterns of play: play-personas in user-centred game development. In: Barry, A., Helen, K., Tanya, K. (eds.) Breaking New Ground: Innovation in Games, Play, Practice and Theory: Proceedings of the 2009 Digital Games Research Association Conference. Brunel University, London, UK (2009)Google Scholar
  20. Champalle, O., Sehaba, K., Cosmas, D., Mille, A., Prié, Y.: Assistance to trainers for the observation and analysis activities of operators trainees on nuclear power plant full-scope simulator. In: International Conference on Intelligent Networking and Collaborative Systems (INCoS 2012), pp. 33–40. IEEE Computer Society, Bucharest, Romania (2012)Google Scholar
  21. Champin, P.A., Prié, Y., Aubert, O., Conil, F., Cram, D.: ktbs: Kernel for trace-based systems. http://kernel-for-trace-based-systems.readthedocs.org/en/latest/, a reference implementation of the notion of Trace Based Management System. Allows to store and compute modeled traces, and access them through a RESTful interface (2011)
  22. Chen, M., Kolko, B.E., Cuddihy, E., Medina, E.: Modeling but not measuring engagement in computer games. In: Proceedings of the 7th International Conference on Games Learning Society, GLS’11, pp. 55–61. ETC Press, Madison, Wisconsin, USA (2011)Google Scholar
  23. Chittaro, L., Zangrando, N.: The persuasive power of virtual reality: effects of simulated human distress on attitudes towards fire safety. In: Proceedings of the 5th International Conference on Persuasive Technology, PERSUASIVE 2010, pp. 58–69. Springer, Copenhagen, Denmark (2010)Google Scholar
  24. Christensen, H., Griffiths, M.K., Farrer, L.: Adherence in internet interventions for anxiety and depression: systematic review. J. Med. Internet Res. 11(2), e13 (2009)Google Scholar
  25. Clauzel, D., Sehaba, K., Prié, Y.: Modelling and visualising traces for reflexivity in synchronous collaborative systems. In: Proceedings of the 2009 International Conference on Intelligent Networking and Collaborative Systems, INCoS 2009, pp. 16–23. IEEE Computer Society, Barcelona, Spain (2009)Google Scholar
  26. Clauzel, D., Sehaba, K., Prié, Y.: Enhancing synchronous collaboration by using interactive visualisation of modelled traces. Simul. Model. Pract. Theory 19(1), 84–97 (2011)CrossRefGoogle Scholar
  27. Cocea, M., Weibelzahl, S.: Log file analysis for disengagement detection in e-learning environments. User Model. User-Adapt. Interact. 19(4), 341–385 (2009)CrossRefGoogle Scholar
  28. Consolvo, S., Everitt, K., Smith, I., Landay, J.A.: Design requirements for technologies that encourage physical activity. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2006, pp. 457–466. ACM, Montréal, Québec, Canada (2006)Google Scholar
  29. D’Mello, S.K., Craig, S.D., Sullins, J., Graesser, A.C.: Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Educ. 16(1), 3–28 (2006)Google Scholar
  30. Dietterich, T.G.: Machine learning for sequential data: a review. In: Proceedings of the Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition, pp. 15–30. Springer, Windsor, ON, Canada (2002)Google Scholar
  31. Doherty, G., Coyle, D., Sharry, J.: Engagement with online mental health interventions: an exploratory clinical study of a treatment for depression. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2012, pp. 1421–1430. ACM, Austin, TX, USA (2012)Google Scholar
  32. Donkin, L., Christensen, H., Naismith, L.S., Neal, B., Hickie, B.I., Glozier, N.: A systematic review of the impact of adherence on the effectiveness of e-therapies. J. Med. Internet Res. 13(3), e52 (2011)CrossRefGoogle Scholar
  33. Engelmann, T., Dehler, J., Bodemer, D., Buder, J.: Knowledge awareness in cscl: a psychological perspective. Comput. Human Behav. 25(4), 949–960 (2009)CrossRefGoogle Scholar
  34. Fogg, B.J.: Persuasive Technology: Using Computers to Change What We Think and Do. Morgan Kaufmann, Amsterdam (2002)Google Scholar
  35. Fogg, B.: A behavior model for persuasive design. In: Proceedings of the 4th International Conference on Persuasive Technology, Persuasive 2009, pp. 40:1–40:7. ACM, Claremont, CA, USA (2009)Google Scholar
  36. Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: potential of the concept, state of the evidence. Rev. Educ. Res. 74, 59–109 (2004)CrossRefGoogle Scholar
  37. Froehlich, J., Dillahunt, T., Klasnja, P., Mankoff, J., Consolvo, S., Harrison, B., Landay, J.A.: Ubigreen: investigating a mobile tool for tracking and supporting green transportation habits. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2009, pp. 1043–1052. ACM, Boston, Massachusetts, USA (2009)Google Scholar
  38. Gagné, A.R., El-Nasr, M.S., Shaw, C.D.: A deeper look at the use of telemetry for analysis of player behavior in rts games. In: Proceedings of the 10th International Conference on Entertainment Computing, ICEC’11, pp. 247–257. Springer, Vancouver, Canada (2011)Google Scholar
  39. Garris, R., Ahlers, R., Driskell, J.E.: Games, motivation, and learning: a research and practice model. Simul. Gaming 33(4), 441–467 (2002)CrossRefGoogle Scholar
  40. Hägni, K., Eng, K., Hepp-Reymond, M.C., Holper, L., Keisker, B., Siekierka, E., Kiper, D.C.: The effect of task and ownership on time estimation in virtual environments. In: Presence 2007: The 10th Annual International Workshop on Presence, pp. 145–150, Barcelona, Spain (2007)Google Scholar
  41. Hassenzahl, M., Tractinsky, N.: User experience—a research agenda. Behav. Inf. Technol. 25(2), 91–97 (2006)CrossRefGoogle Scholar
  42. Honka, A., Kaipainen, K., Hietala, H., Saranummi, N.: Rethinking health: Ict-enabled services to empower people to manage their health. IEEE Rev. Biomed. Eng. 4, 119–139 (2011)CrossRefGoogle Scholar
  43. Jackson, T.: Motivating Sustainable Consumption. A Review of Evidence on Consumer Behaviour and Behavioural Change. Tech. rep., University of Surrey, http://hiveideas.com/attachments/044_motivatingscfinal_000.pdf, a report to the Sustainable Development Research Network (2005)
  44. Janssen, J., Erkens, G., Kirschner, P.A.: Group awareness tools: it’s what you do with it that matters. Comput. Hum. Behav. 27(3), 1046–1058 (2011)CrossRefGoogle Scholar
  45. Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and defining the experience of immersion in games. Int. J. Hum.–Comput. Stud. 66(9), 641–661 (2008)CrossRefGoogle Scholar
  46. Kaptelinin, V.: The object of activity: making sense of the sense-maker. Mind Cult. Activity 12(1), 4–18 (2005)CrossRefGoogle Scholar
  47. Kaptelinin, V., Nardi, B.A.: Acting with Technology: Activity Theory and Interaction Design. MIT Press, Cambridge (2006)Google Scholar
  48. Kim, J.H., Gunn, D.V., Schuh, E., Phillips, B., Pagulayan, R.J., Wixon, D.: Tracking real-time user experience (true): a comprehensive instrumentation solution for complex systems. In: Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, CHI 2008, pp. 443–452. ACM, Florence, Italy (2008)Google Scholar
  49. Kivikangas, J.M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., Ravaja, N.: A review of the use of psychophysiological methods in game research. J. Gaming Virtual Worlds 3(3), 181–199 (2011)CrossRefGoogle Scholar
  50. Kjeldskov, J., Skov, M.B., Paay, J., Pathmanathan, R.: Using mobile phones to support sustainability: a field study of residential electricity consumption. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2012, pp. 2347–2356. ACM, Austin, TX, USA (2012)Google Scholar
  51. Köck, M., Paramythis, A.: Activity sequence modelling and dynamic clustering for personalized e-learning. User Model. User-Adapt. Interact. 21(1–2), 51–97 (2011)CrossRefGoogle Scholar
  52. Law, E.L.C., Roto, V., Hassenzahl, M., Vermeeren, A.P., Kort, J.: Understanding, scoping and defining user experience: a survey approach. In: Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, pp. 719–728. ACM, Boston, MA, USA (2009)Google Scholar
  53. Leontiev, A.N.: Activity, Consciousness and Personality. Prentice Hall, Englewood Cliffs (1978)Google Scholar
  54. Linehan, C., Kirman, B., Lawson, S., Chan, G.: Practical, appropriate, empirically-validated guidelines for designing educational games. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2011, pp. 1979–1988. ACM, Vancouver, BC, Canada (2011)Google Scholar
  55. Maitland, J., Chalmers, M.: Designing for peer involvement in weight management. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2011, pp. 315–324. ACM, Vancouver, BC, Canada (2011)Google Scholar
  56. Mandryk, R.L., Atkins, M.S., Inkpen, K.M. A continuous and objective evaluation of emotional experience with interactive play environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2006, pp. 1027–1036. ACM, Montréal, QC, Canada (2006)Google Scholar
  57. Michel, C., Lavoué, E., Pietrac, L.: A dashboard to regulate project-based learning. In: Proceedings of the 7th European Conference on Technology Enhanced Learning, EC-TEL 2012, pp. 250–263. Springer, Saarbrücken, Germany (2012)Google Scholar
  58. Michie, S., Johnston, M., Francis, J., Hardeman, W., Eccles, M.: From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques. Appl. Psychol. 57(4), 660–680 (2008)CrossRefGoogle Scholar
  59. Mintz, J., Aagaard, M.: The application of persuasive technology to educational settings. Educ. Technol. Res. Dev. 60(3), 483–499 (2012)CrossRefGoogle Scholar
  60. O’Brien, H.L., Toms, E.G.: What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inf. Sci. 59(6), 938–955 (2008)CrossRefGoogle Scholar
  61. Orji, R., Vassileva, J., Mandryk, R.L.: Modeling the efficacy of persuasive strategies for different gamer types in serious games for health. User Model. User-Adapt. Interact. (this issue) (2014)Google Scholar
  62. Pagulayan, R.J., Keeker, K., Wixon, D., Romero, R.L., Fuller, T.: User-centered design in games. In: Jacko, J.A., Sears, A. (eds.) The Human–Computer Interaction Handbook, pp. 883–906. Lawrence Erlbaum, Hillsdale (2003)Google Scholar
  63. Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)Google Scholar
  64. Rich, C., Ponsleur, B., Holroyd, A., Sidner, C.L.: Recognizing engagement in human–robot interaction. In: Proceedings of the 5th ACM/IEEE International Conference on Human–Robot Interaction, HRI ’10, pp. 375–382. IEEE Press, Osaka, Japan (2010)Google Scholar
  65. Ritterband, L., Thorndike, F., Cox, D., Kovatchev, B., Gonder-Frederick, L.: A behavior change model for internet interventions. Ann. Behav. Med. 38(1), 18–27 (2009)Google Scholar
  66. Ryan, R.M., Deci, E.L.: Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55, 68–78 (2000)CrossRefGoogle Scholar
  67. Scheffel, M., Wolpers, M., Beer, F.: Analyzing contextualized attention metadata with rough set methodologies to support self-regulated learning. In: 2010 IEEE 10th International Conference on Advanced Learning Technologies (ICALT), ICALT 2010, pp. 125–129. IEEE Computer Society, Sousse, Tunisia (2010)Google Scholar
  68. Schön, D.A.: The Reflective Practitioner: How Professionals Think in Action. Basic Books (1984)Google Scholar
  69. Schubert, T., Friedmann, F., Regenbrecht, H.: The experience of presence: factor analytic insights. Presence: Teleoper. Virtual Environ. 10(3), 266–281 (2001)CrossRefGoogle Scholar
  70. Settouti, L.S., Prié, Y., Marty, J.C., Mille, A.: A trace-based system for technology-enhanced learning systems personalisation. In: Proceedings of the 2009 Ninth IEEE International Conference on Advanced Learning Technologies, ICALT 2009, pp. 93–97. IEEE Computer Society, Riga, Latvia (2009)Google Scholar
  71. Sidner, C.L., Kidd, C.D., Lee, C., Lesh, N.: Where to look: a study of human–robot engagement. In: Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI ’04, pp. 78–84. ACM, Funchal, Madeira, Portugal (2004)Google Scholar
  72. Tychsen ,A., Canossa, A: Defining personas in games using metrics. In: Proceedings of the 2008 Conference on Future Play: Research, Play, Share, Future Play 2008, pp. 73–80. ACM, Toronto, ON, Canada (2008)Google Scholar
  73. Vassileva, J.: Motivating participation in social computing applications: a user modeling perspective. User Model. User-Adapt. Interact. 22(1–2), 177–201 (2012)MathSciNetCrossRefGoogle Scholar
  74. Vygotsky, L.: Mind in Society: The Development of Higher Psychological Processes. Harvard University Press, Cambridge (1978)Google Scholar
  75. Zimmerman, B.J.: Attaining Self-Regulation: A Social Cognitive Perspective, pp. 13–40 (2000)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  1. 1.Université de Lyon, CNRSUniversité Lyon 1, LIRIS, UMR5205VilleurbanneFrance
  2. 2.Université de Lyon, CNRSUniversité Lyon 2, LIRIS, UMR5205Bron CedexFrance
  3. 3.Magellan, IAE LyonUniversité Jean Moulin Lyon 3Lyon Cedex 08France

Personalised recommendations