Skip to main content
SpringerLink
Log in

Towards Understanding the Nonverbal Signatures of Engagement in Super Mario Bros

  • Conference paper
Book cover User Modeling, Adaptation, and Personalization (UMAP 2014)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 8538))

Abstract

In this paper, we present an approach for predicting users’ level of engagement from nonverbal cues within a game environment. We use a data corpus collected from 28 participants (152 minutes of video recording) playing the popular platform game Super Mario Bros. The richness of the corpus allows extraction of several visual and facial expression features that were utilised as indicators of players’ affects as captured by players’ self-reports. Neuroevolution preference learning is used to construct accurate models of player experience that approximate the relationship between extracted features and reported engagement. The method is supported by a feature selection technique for choosing the relevant subset of features. Different setup settings were implemented to analyse the impact of the type of the features and the position of the extraction window on the modelling accuracy. The results obtained show that highly accurate models can be constructed (with accuracies up to 96.82%) and that players’ nonverbal behaviour towards the end of the game is the most correlated with engagement. The framework presented is part of a bigger picture where the generated models are utilised to tailor content generation to a player’s particular needs and playing characteristics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abadi, M.K., Staiano, J., Cappelletti, A., Zancanaro, M., Sebe, N.: Multimodal engagement classification for affective cinema. In: Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 411–416. IEEE (2013)

    Google Scholar 

  2. Asteriadis, S., Tzouveli, P., Karpouzis, K., Kollias, S.: Estimation of behavioral user state based on eye gaze and head poseapplication in an e-learning environment. Multimedia Tools and Applications 41(3), 469–493 (2009)

    Article  Google Scholar 

  3. Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., Karpouzis, K.: Multimodal emotion recognition from expressive faces, body gestures and speech. In: Artificial Intelligence and Innovations 2007: From Theory to Applications, pp. 375–388. Springer (2007)

    Google Scholar 

  4. Castellano, G., Pereira, A., Leite, I., Paiva, A., McOwan, P.W.: Detecting user engagement with a robot companion using task and social interaction-based features. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 119–126. ACM (2009)

    Google Scholar 

  5. Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction 19(3), 267–303 (2009)

    Article  Google Scholar 

  6. Mello, D., Craig, S.K., Graesser, S.D., Multi-method, A.C.: assessment of affective experience and expression during deep learning. Int. J. Learn. Technol. 4(3/4), 165–187 (2009)

    Article  Google Scholar 

  7. Fürnkranz, J., Hüllermeier, E.: Pairwise preference learning and ranking. In: Lavrač, N., Gamberger, D., Todorovski, L., Blockeel, H. (eds.) ECML 2003. LNCS (LNAI), vol. 2837, pp. 145–156. Springer, Heidelberg (2003)

    Google Scholar 

  8. Grafsgaard, J.F., Boyer, K.E., Lester, J.C.: Toward a machine learning framework for understanding affective tutorial interaction. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 52–58. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  9. Ioannou, S., Caridakis, G., Karpouzis, K., Kollias, S.: Robust feature detection for facial expression recognition. Journal on Image and Video Processing (2) (2007)

    Google Scholar 

  10. Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies 66(9), 641–661 (2008)

    Article  Google Scholar 

  11. Kapoor, A., Burleson, W., Picard, R.: Automatic prediction of frustration. International Journal of Human-Computer Studies (8), 724–736 (2007)

    Google Scholar 

  12. Karpouzis, K., Shaker, N., Yannakakis, G.N., Asteriadis, S.: The platformer experience dataset. IEEE Transactions on Computational Intelligence and AI in Games (2014)

    Google Scholar 

  13. Küblbeck, C., Ernst, A.: Face detection and tracking in video sequences using the modifiedcensus transformation. Image and Vision Computing 24(6), 564–572 (2006)

    Article  Google Scholar 

  14. Lisetti, C.L., Nasoz, F.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing 2004, 1672–1687 (2004)

    Article  Google Scholar 

  15. Mandryk, R., Inkpen, K., Calvert, T.: Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology (2), 141–158 (2006)

    Google Scholar 

  16. Martinez, H., Yannakakis, G.: Mining multimodal sequential patterns: A case study on affect detection. In: Proceedings of the 13th International Conference in Multimodal Interaction, ICMI 2011, Alicante. ACM Press (November 2011)

    Google Scholar 

  17. McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., Graesser, A.: Facial features for affective state detection in learning environments. In: Proceedings of the 29th Annual Cognitive Science Society, pp. 467–472 (2007)

    Google Scholar 

  18. Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling player experience for content creation. IEEE Transactions on Computational Intelligence and AI in Games 2(1), 54–67 (2010)

    Article  Google Scholar 

  19. Peters, C., Asteriadis, S., Karpouzis, K., de Sevin, E.: Towards a real-time gaze-based shared attention for a virtual agent. In: Workshop on Affective Interaction in Natural Environments (AFFINE), ACM International Conference on Multimodal Interfaces, ICMI 2008 (2008)

    Google Scholar 

  20. Shaker, N., Yannakakis, G.N., Togelius, J.: Towards Automatic Personalized Content Generation for Platform Games. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE (2010)

    Google Scholar 

  21. Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: A game-based corpus for analysing the interplay between game context and player experience. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part II. LNCS, vol. 6975, pp. 547–556. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  22. Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: Fusing visual and behavioral cues for modeling user experience in games. IEEE Transactions on Cybernetics 43(6), 1519–1531 (2013)

    Article  Google Scholar 

  23. Shaker, N., Togelius, J., Yannakakis, G.N., Weber, B., Shimizu, T., Hashiyama, T., Sorenson, N., Pasquier, P., Mawhorter, P., Takahashi, G., Smith, G., Baumgarten, R.: The 2010 Mario AI championship: Level generation track. IEEE Transactions on Computational Intelligence and Games (2011)

    Google Scholar 

  24. Shaker, N., Yannakakis, G.N., Togelius, J.: Crowd-sourcing the aesthetics of platform games. IEEE Transactions on Computational Intelligence and AI in Games (2013)

    Google Scholar 

  25. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing 3(1), 42–55 (2012)

    Article  Google Scholar 

  26. Tognetti, S., Garbarino, M., Bonanno, A.T., Matteucci, M., Bonarini, A.: Enjoyment recognition from physiological data in a car racing game. In: Proceedings of the 3rd international Workshop on Affective Interaction in Natural Environments, pp. 3–8. ACM (2010)

    Google Scholar 

  27. Yannakakis, G.N., Togelius, J.: Experience-Driven Procedural Content Generation. IEEE Transactions on Affective Computing (2011)

    Google Scholar 

  28. Yannakakis, G.N., Hallam, J.: Entertainment modeling through physiology in physical play. Int. J. Hum.-Comput. Stud. 66, 741–755 (2008)

    Google Scholar 

  29. Yannakakis, G.N., Maragoudakis, M., Hallam, J.: Preference learning for cognitive modeling: a case study on entertainment preferences. Trans. Sys. Man Cyber. Part A 39, 1165–1175 (2009)

    Google Scholar 

  30. Yannakakis, G.N., Maragoudakis, M., Hallam, J.: Preference learning for cognitive modeling: a case study on entertainment preferences. Trans. Sys. Man Cyber. Part A 39, 1165–1175 (November 2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IT University of Copenhagen, Rued Langaards Vej 7, 2300, Copenhagen, Denmark

    Noor Shaker

  2. Independent Researcher, Damascus, Syria

    Mohammad Shaker

Authors
  1. Noor Shaker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Shaker
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computing, University of Leeds, United Kingdom

    Vania Dimitrova

  2. University of Haifa, Israel

    Tsvi Kuflik

  3. Department of Information and Computer Sciences, University of Hawaii, 96822, Honolulu, HI, USA

    David Chin

  4. Free University of Bozen - Bolzano, Piazza Domenicani 3, 39100, Bolzano, Italy

    Francesco Ricci

  5. Department of Computer Science, Aalborg University, Selma Lagerlofs Vej, 300 9220, Aalborg-East, Denmark

    Peter Dolog

  6. EEMCS - Web Information Systems, 2600 GA, Delft, The Netherlands

    Geert-Jan Houben

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Shaker, N., Shaker, M. (2014). Towards Understanding the Nonverbal Signatures of Engagement in Super Mario Bros. In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, GJ. (eds) User Modeling, Adaptation, and Personalization. UMAP 2014. Lecture Notes in Computer Science, vol 8538. Springer, Cham. https://doi.org/10.1007/978-3-319-08786-3_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-08786-3_38

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-08785-6

  • Online ISBN: 978-3-319-08786-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation