Skip to main content
SpringerLink
Log in

I Feel I Feel You: A Theory of Mind Experiment in Games

  • Technical Contribution
  • Published:
KI - Künstliche Intelligenz Aims and scope Submit manuscript

Abstract

In this study into the player’s emotional theory of mind (ToM) of gameplaying agents, we investigate how an agent’s behaviour and the player’s own performance and emotions shape the recognition of a frustrated behaviour. We focus on the perception of frustration as it is a prevalent affective experience in human-computer interaction. We present a testbed game tailored towards this end, in which a player competes against an agent with a frustration model based on theory. We collect gameplay data, an annotated ground truth about the player’s appraisal of the agent’s frustration, and apply face recognition to estimate the player’s emotional state. We examine the collected data through correlation analysis and predictive machine learning models, and find that the player’s observable emotions are not correlated highly with the perceived frustration of the agent. This suggests that our subject’s ToM is a cognitive process based on the gameplay context. Our predictive models—using ranking support vector machines—corroborate these results, yielding moderately accurate predictors of players’ ToM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. http://plt.institutedigitalgames.com.

  2. The best (C) and RBF\(\gamma \) values for each model are:\(\mu _A \rightarrow \mu _A\): game: (0.1) 0.5; facial: (100) 0.1; all: (1) 0.01.\(\hat{A} \rightarrow \hat{A}\): game: (0.1) 0.5; facial: (10) 1; all: (0.1) 0.1.\(\hat{A} \rightarrow \mu _A\): game: (0.1) 0.1; facial: (0.1) 0.01; all: (0.1) 0.5.\(\mu _A \rightarrow \hat{A}\): game: (0.5) 0.01; facial: (0.1) 0.01; all: (0.5) 0.01.

References

  1. Albrecht SV, Stone P (2018) Autonomous agents modelling other agents: a comprehensive survey and open problems. Artif Intell 258:66–95

    Article  MathSciNet  MATH  Google Scholar 

  2. Amsel A (1992) Frustration theory: an analysis of dispositional learning and memory. Cambridge University Press, Cambridge

    Book  Google Scholar 

  3. Arrabales R, Ledezma A, Sanchis A (2009) Towards conscious-like behavior in computer game characters. In: Proceedings of 2009 IEEE symposium on computational intelligence and games. IEEE, Milan, Italy, pp 217–224. https://doi.org/10.1109/CIG.2009.5286473

    Chapter  Google Scholar 

  4. Berkowitz L (1989) Frustration-aggression hypothesis: examination and reformulation. Psychol Bull 106(1):59–73

    Article  Google Scholar 

  5. Bessiere K, Newhagen JE, Robinson JP, Shneiderman B (2006) A model for computer frustration: the role of instrumental and dispositional factors on incident, session, and post-session frustration and mood. Comput Hum Behav 22(6):941–961

    Article  Google Scholar 

  6. Bormann D, Greitemeyer T (2015) Immersed in virtual worlds and minds: effects of in-game storytelling on immersion, need satisfaction, and affective theory of mind. Soc Psychol Personal Sci 6(6):646–652

    Article  Google Scholar 

  7. Bruner JS (1981) Intention in the structure of action and interaction. Adv Infancy Res 1:41–56

    Google Scholar 

  8. Camilleri E, Yannakakis GN, Liapis A (2017) Towards general models of player affect. In: Proceedings of 2017 seventh international conference on affective computing and intelligent interaction (ACII). IEEE, San Antonio, TX, pp 333–339. https://doi.org/10.1109/ACII.2017.8273621

    Chapter  Google Scholar 

  9. Canossa A, Drachen A, Sørensen JRM (2011) Arrrgghh!!!: blending quantitative and qualitative methods to detect player frustration. In: Proceedings of the 6th international conference on foundations of digital games (FDG ’11). Association for Computing Machinery, New York, NY, pp 61–68. https://doi.org/10.1145/2159365.2159374

    Chapter  Google Scholar 

  10. Carver CS, Scheier MF (2012) Attention and self-regulation: a control-theory approach to human behavior. Springer, Berlin

    Google Scholar 

  11. Critchley HD, Corfield D, Chandler M, Mathias C, Dolan RJ (2000) Cerebral correlates of autonomic cardiovascular arousal: a functional neuroimaging investigation in humans. J Physiol 523(1):259–270

    Article  Google Scholar 

  12. Damasio AR (1996) The somatic marker hypothesis and the possible functions of the prefrontal cortex. Philos Trans R Soc Lond Ser B 351(1346):1413–1420

    Article  Google Scholar 

  13. De Weerd H, Verbrugge R, Verheij B (2015) Higher-order theory of mind in the tacit communication game. Biol Inspired Cogn Archit 11:10–21

    Google Scholar 

  14. Dunn BD, Dalgleish T, Lawrence AD (2006) The somatic marker hypothesis: a critical evaluation. Neurosci Biobehav Rev 30(2):239–271

    Article  Google Scholar 

  15. Ekman P, Freisen WV, Ancoli S (1980) Facial signs of emotional experience. J Person Soc Psychol 39(6):1125–1134

    Article  Google Scholar 

  16. Farrugia VE, Martínez HP, Yannakakis GN (2015) The preference learning toolbox, p 3. arXiv:150601709

  17. Fernández-Dols JM (2013) Advances in the study of facial expression: an introduction to the special section. Emot Rev 5(1):3–7

    Article  Google Scholar 

  18. Fredricks GA, Nelsen RB (2007) On the relationship between Spearman’s rho and Kendall’s tau for pairs of continuous random variables. J Stat Plan Inference 137(7):2143–2150

    Article  MathSciNet  MATH  Google Scholar 

  19. Fürnkranz J, Hüllermeier E (2003) Pairwise preference learning and ranking. In: Lavrač N, Gamberger D, Blockeel H, Todorovski L (eds) Machine Learning: ECML 2003, vol 2837. Lecture Notes in Computer Science. Springer, Berlin, pp 145–156. https://doi.org/10.1007/978-3-540-39857-8_15

    Chapter  MATH  Google Scholar 

  20. Gallagher HL, Frith CD (2003) Functional imaging of ‘theory of mind’. Trends Cogn Sci 7(2):77–83

    Article  Google Scholar 

  21. Garfield JL, Peterson CC, Perry T (2001) Social cognition, language acquisition and the development of the theory of mind. Mind Lang 16(5):494–541

    Article  Google Scholar 

  22. Gilleade KM, Dix A (2004) Using frustration in the design of adaptive videogames. In: Proceedings of the ACM SIGCHI international conference on advances in computer entertainment technology (ACE ’04). Association for Computing Machinery, New York, NY, pp 228–232. https://doi.org/10.1145/1067343.1067372

    Chapter  Google Scholar 

  23. Goodie AS, Doshi P, Young DL (2012) Levels of theory-of-mind reasoning in competitive games. J Behav Decis Mak 25(1):95–108

    Article  Google Scholar 

  24. Gopnik A, Wellman HM (1992) Why the child’s theory of mind really is a theory. Mind Lang 7(1–2):145–171

    Article  Google Scholar 

  25. Hebb DO (1955) Drives and the CNS (conceptual nervous system). Psychol Rev 62(4):243–254

    Article  Google Scholar 

  26. Hedden T, Zhang J (2002) What do you think i think you think? Strategic reasoning in matrix games. Cognition 85(1):1–36

    Article  Google Scholar 

  27. Joachims T (2002) Optimizing search engines using clickthrough data. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining (KDD ’02). Association for Computing Machinery, New York, NY, pp 133–142. https://doi.org/10.1145/775047.775067

    Chapter  Google Scholar 

  28. Lankoski P, Björk S (2007) Gameplay design patterns for believable non-player characters. In: Proceedings of the 2007 DiGRA international conference: Situated Play, University of Yokyo, pp 416–423. ISSN 2342-9666

  29. Lopes P, Yannakakis GN, Liapis A (2017) Ranktrace: relative and unbounded affect annotation. In: Proceedings of 2017 seventh international conference on affective computing and intelligent interaction (ACII). IEEE, San Antonio, TX, pp 158–163. https://doi.org/10.1109/ACII.2017.8273594

    Chapter  Google Scholar 

  30. Mariooryad S, Busso C (2013) Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations. In: Proceedings of the 2013 humaine association conference on affective computing and intelligent interaction (ACII ’13). IEEE Computer Society, USA, pp 85–90. https://doi.org/10.1109/ACII.2013.21

    Chapter  Google Scholar 

  31. Martinez H, Yannakakis G, Hallam J (2014) Don’t classify ratings of affect; rank them!. IEEE Trans Affect Comput 1:1–1

    Google Scholar 

  32. McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby Re (2016) Affdex SDK: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems (CHI EA ’16). Association for Computing Machinery, New York, NY, pp 3723–3726. https://doi.org/10.1145/2851581.2890247

    Chapter  Google Scholar 

  33. Meijering B, Van Rijn H, Taatgen N, Verbrugge R (2011) I do know what you think i think: second-order theory of mind in strategic games is not that difficult. Proc Annu Meet Cogn Sci Soc 33:2486–2491

    Google Scholar 

  34. Michel P, El Kaliouby R (2003) Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th international conference on multimodal interfaces (ICMI ’03). Association for Computing Machinery, New York, NY, pp 258–264. https://doi.org/10.1145/958432.958479

    Chapter  Google Scholar 

  35. Nelsen R (2001) Kendall tau metric. Encycl Math 3:226–227

    Google Scholar 

  36. Ortony A, Clore GL, Collins A (1990) The cognitive structure of emotions. Cambridge University Press, Cambridge

    Google Scholar 

  37. Perner J, Wimmer H (1985) “John thinks that Mary thinks that” attribution of second-order beliefs by 5-to 10-year-old children. J Exp Child Psychol 39(3):437–471

    Article  Google Scholar 

  38. Poletti M, Enrici I, Adenzato M (2012) Cognitive and affective theory of mind in neurodegenerative diseases: neuropsychological, neuroanatomical and neurochemical levels. Neurosci Biobehav Rev 36(9):2147–2164

    Article  Google Scholar 

  39. Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125

    Article  Google Scholar 

  40. Premack D, Woodruff G (1978) Does the chimpanzee have a theory of mind? Behav Brain Sci 1(4):515–526

    Article  Google Scholar 

  41. Rabinowitz NC, Perbet F, Song HF, Zhang C, Eslami S, Botvinick M (2018) Machine theory of mind, p 21. arXiv:180207740

  42. Rauterberg M (1995) About a framework for international and information processing of learning systems. In: Falkenberg ED, Hesse W, Olivé A (eds) Proceedings of the IFIP international working conference on information system concepts: towards a consolidation of views. Chapman & Hall, Ltd., London, pp 54–69

    Chapter  Google Scholar 

  43. Roohi S, Takatalo J, Kivikangas JM, Hämäläinen P (2018) Neural network based facial expression analysis of gameevents: a cautionary tale. In: Proceedings of the 2018 annual symposium on computer–human interaction in Play (CHI Play ’18). Association for Computing Machinery, New York, NY, pp 429–437. https://doi.org/10.1145/3242671.3242701

    Chapter  Google Scholar 

  44. Schaafsma SM, Pfaff DW, Spunt RP, Adolphs R (2015) Deconstructing and reconstructing theory of mind. Trends Cogn Sci 19(2):65–72

    Article  Google Scholar 

  45. Sebastian CL, Fontaine NM, Bird G, Blakemore SJ, De Brito SA, McCrory EJ, Viding E (2011) Neural processing associated with cognitive and affective theory of mind in adolescents and adults. Soc Cogn Affect Neurosci 7(1):53–63

    Article  Google Scholar 

  46. Shamay-Tsoory SG, Harari H, Aharon-Peretz J, Levkovitz Y (2010) The role of the orbitofrontal cortex in affective theory of mind deficits in criminal offenders with psychopathic tendencies. Cortex 46(5):668–677

    Article  Google Scholar 

  47. Vapnik V (1995) Chapter 5 constructing learning algorithms. In: The nature of statistical learning theory. Springer, New York, NY, pp 119–157. https://doi.org/10.1007/978-1-4757-2440-0_6

    Chapter  MATH  Google Scholar 

  48. Weilbächer RA, Gluth S (2016) The interplay of hippocampus and ventromedial prefrontal cortex in memory-based decision making. Brain Sci 7(4):15

    Google Scholar 

  49. Yannakakis GN, Martinez HP (2015) Grounding truth via ordinal annotation. In: Proceedings of the 2015 international conference on affective computing and intelligent interaction (ACII) (ACII ’15). IEEE Computer Society, USA, pp 574–580. https://doi.org/10.1109/ACII.2015.7344627

    Chapter  Google Scholar 

  50. Yannakakis GN, Martínez HP (2015) Ratings are overrated!. Front ICT 2:13

    Article  Google Scholar 

  51. Yannakakis GN, Togelius J (2018) Artificial intelligence and games. Springer Nature, New York

    Book  Google Scholar 

  52. Yannakakis GN, Cowie R, Busso C (2017) The ordinal nature of emotions. In: Proceedings of the 2017 international conference on affective computing and intelligent interaction (ACII). IEEE, San Antonio, TX, pp 248–255. https://doi.org/10.1109/ACII.2017.8273608

    Chapter  Google Scholar 

  53. Yannakakis GN, Cowie R, Busso C (2018) The ordinal nature of emotions: an emerging approach. In: IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2018.2879512

  54. Yerkes RM, Dodson JD (1908) The relation of strength of stimulus to rapidity of habit-formation. J Comp Neurol Psychol 18(5):459–482

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Digital Games, University of Malta, Msida, MSD, 2080, Malta

    David Melhart, Georgios N. Yannakakis & Antonios Liapis

Authors
  1. David Melhart
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Georgios N. Yannakakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonios Liapis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Melhart.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Melhart, D., Yannakakis, G.N. & Liapis, A. I Feel I Feel You: A Theory of Mind Experiment in Games. Künstl Intell 34, 45–55 (2020). https://doi.org/10.1007/s13218-020-00641-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13218-020-00641-2

Keywords

Search

Navigation