Emotion and Body-Based Games: Overview and Opportunities

  • Nadia Bianchi-Berthouze
  • Katherine Isbister
Part of the Socio-Affective Computing book series (SAC, volume 4)


In this chapter we examine research and theory concerning body movement as a means for expressing emotion, and techniques for recognizing expressions of emotion in the context of games. We discuss body movement as a means for biasing emotional experience and encouraging bonding in social interaction. Finally, we discuss gaps and opportunities for future research. Promising directions include broadening the scope of body-based games and emotion to take proprioception into account, as well as other less explored body channels such as muscle activation and action-related sound.


Body movement Embodied affect Game design Exergames Exertion games Body games Movement-based games Emotion 


  1. 1.
    Damasio AR (1994) Descartes’ error: emotion, reason and the human brain. Avon Books, New YorkGoogle Scholar
  2. 2.
    de Gelder B (2009) Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philos Trans R Soc 364(3):3475–3484CrossRefGoogle Scholar
  3. 3.
    Sinclair J, Hingston P, Masek M (2007) Considerations for the design of exergames. Proc GRAPHITE’07 1:289–295Google Scholar
  4. 4.
    Mueller FF, Edge D, Vetere F, Gibbs MR, Agamanolis S, Bongers B, Sheridan JG (2011) Designing sports: a framework for exertion games. In: Proceedings of CHI 2011, pp 2651–2660Google Scholar
  5. 5.
    Márquez Segura E, Waern A, Moen J, Johansson C (2013) The design space of body games: technological, physical, and social design. In: Proceedings of CHI 2013, Paris, FranceGoogle Scholar
  6. 6.
    Isbister K, Mueller F (2015) Guidelines for the design of movement-based games and their relevance to HCI. Hum Comput Interact, Spec Issue Games HCI 30(3–4):366–399CrossRefGoogle Scholar
  7. 7.
    Kleinginna PR, Kleinginna AM (1981) A categorized list of emotion definitions with suggestions for a consensual definition. Motiv Emot 5:345–379CrossRefGoogle Scholar
  8. 8.
    Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):693–727CrossRefGoogle Scholar
  9. 9.
    Kleinsmith A, Bianchi-Berthouze N (2013) Affective body expression perception and recognition: a survey. IEEE Trans Affect Comput 4(1):15–33CrossRefGoogle Scholar
  10. 10.
    Ekman P, Friesen W (1967) Head and body cues in the judgment of emotion: a reformulation. Percept Mot Skills 24:711–724CrossRefPubMedGoogle Scholar
  11. 11.
    Paterson HM, Pollick FE, Sanford AJ (2001) The role of velocity in affect discrimination. In: Proceedings of 23rd annual conference of the cognitive science society, Lawrence Erlbaum Associates, pp 756–761Google Scholar
  12. 12.
    Kleinsmith A, Bianchi-Berthouze N (2005) Grounding affective dimensions into posture features. LNCS: Proceedings of 1st international conference on on affective computing and intelligent interaction, pp 263–270Google Scholar
  13. 13.
    Kleinsmith A, Bianchi-Berthouze N, Steed A (2011) Automatic recognition of non-acted affective postures. IEEE Trans Syst Man Cybern Part B 41(4):1027–1038CrossRefGoogle Scholar
  14. 14.
    Karg M, Kuhnlenz K, Buss M (2010) Recognition of affect based on gait patterns. IEEE Trans Syst Man Cybern Part B 40(4):1050–1061CrossRefGoogle Scholar
  15. 15.
    Kleinsmith A, De Silva R, Bianchi-Berthouze N (2006) Cross-cultural differences in recognizing affect from body posture. Interact Comput 18(6):1371–1389CrossRefGoogle Scholar
  16. 16.
    Coulson M (2004) Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. J Nonverbal Behav 28:117–139CrossRefGoogle Scholar
  17. 17.
    Wallbott HG (1998) Bodily expression of emotion. Eur J Soc Psychol 28:879–896CrossRefGoogle Scholar
  18. 18.
    Aviezer H, Trope Y, Todorov A (2012) Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 338:1225–1229, Issue 30CrossRefPubMedGoogle Scholar
  19. 19.
    Lange J, Lappe M (2007) The role of spatial and temporal information in biological motion perception. Adv Cogn Psychol 3(4):419–428CrossRefGoogle Scholar
  20. 20.
    Atkinson AP, Dittrich WH, Gemmell AJ, Young AW (2007) Evidence for distinct contributions of form and motion information to the recognition of emotions from body gestures. Cognition 104(1):59–72CrossRefPubMedGoogle Scholar
  21. 21.
    Roether C, Omlor L, Christensen A, Giese MA (2009) Critical features for the perception of emotion from gait. J Vis 8(6):1–32, No 15Google Scholar
  22. 22.
    Laban R (1980) The mastery of movement. Plymouth, UKGoogle Scholar
  23. 23.
    Bianchi-Berthouze N, Kleinsmith A (2003) A categorical approach to affective gesture recognition. Connect Sci 15(4):259–269CrossRefGoogle Scholar
  24. 24.
    Dael N, Bianchi-Berthouze N, Kleinsmith N, Mohr C (2015) Measuring body movement: current and future directions in proxemics and kinesics. In: The APA handbook of nonverbal communication. APAGoogle Scholar
  25. 25.
    Fourati N, Pelachaud C (2015) Multi-level classification of emotional body expression. In: International conference on automatic face and gesture recognition, pp 1–8.  10.1109/FG.2015.7163145
  26. 26.
    Fourati N, Pelachaud C (2015) Relevant body cues for the classification of emotional body expression in daily actions. In: IEEE proceedings of international conference on affective computing & intelligent interactionGoogle Scholar
  27. 27.
    Kleinsmith A, Bianchi-Berthouze N (2007) Recognizing affective dimensions from body posture, vol 4738, Lecture Notes in Computer Science. LNCS. Springer, Berlin, pp 48–58Google Scholar
  28. 28.
    Isbister K (2012) How to stop being a buzzkill: designing Yamove!, a mobile tech mash-up to truly augment social play. Proceedings of MobileHCIGoogle Scholar
  29. 29.
    Camurri A, Mazzarino B, Volpe G (2004) Analysis of expressive gesture: the EyesWeb expressive gesture processing library. Gesture-Based Commun Hum-Comput Interact LNCS 2915:460–467CrossRefGoogle Scholar
  30. 30.
    Zacharatos H, Gatzoulis C, Chrysanthou Y, Aristidou A (2014) Automatic emotion recognition based on body movement analysis: a survey. IEEE Comput Graph Appl 34(6):35–45CrossRefPubMedGoogle Scholar
  31. 31.
    Kleinsmith A, Bianchi-Berthouze N (2011) Form as a cue in the automatic recognition of non-acted affective body expressions, vol 6975, LNCS. Springer, Memphis, pp 155–164Google Scholar
  32. 32.
    Savva N, Scarinzi A, Bianchi-Berthouze N (2012) Continuous recognition of player’s affective body expression as dynamic quality of aesthetic experience. IEEE Transact Comput Intell AI Games 4(3):199–212CrossRefGoogle Scholar
  33. 33.
    Zacharatos H, Gatzoulis C, Chrysanthou Y, Aristidou A (2013a) Emotion recognition for exergames using Laban Movement analysis. In Proceedings of Motion on Games (MIG ’13). ACM, New York, Article 39Google Scholar
  34. 34.
    Zacharatos H, Gatzoulis C, Chrysanthou G (2013b) Affect recognition during active game playing based on posture skeleton data, In: 8th international conference on computer graphics theory and applicationsGoogle Scholar
  35. 35.
    Aung MS, Bianchi-Berthouze N, Watson P, C de C Williams A (2014) Automatic recognition of fear-avoidance behavior in chronic pain physical rehabilitation. PervasiveHealth ’14 Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare, pp 158–161Google Scholar
  36. 36.
    Aung MSH, Kaltwang S, Romera-Paredes B, Martinez B, Singh A, Cella M, Valstar M, Meng H, Kemp A, Shafizadeh M, Elkins AC, Kanakam N, de Rothschild A, Tyler N, Watson PJ, Williams AC de C, Pantic M, Bianchi-Berthouze N (n.d.) The automatic detection of chronic pain-related expression: requirements, challenges and a multimodal dataset in IEEE Trans. Affective Computing (in press)Google Scholar
  37. 37.
    Olugbade T, Aung MSH, C de C Williams A, Bianchi-Berthouze N (2014) Bi-modal detection of painful reaching for chronic pain rehabilitation systems. ICMI’14, pp 455–458Google Scholar
  38. 38.
    Olugbade TA, Bianchi-Berthouze N, Marquardt N, Williams A CdeC (2015) Pain level recognition using kinematics and muscle activity for physical rehabilitation in chronic pain. IEEE proceedings of international conference on affective computing & intelligent interaction, pp 243–249Google Scholar
  39. 39.
    Griffin HJ, Aung MSH, Romera-Paredes B, McKeown G, Curran W, McLoughlin C, Bianchi-Berthouze N (2013) Laughter type recognition from whole body motion. In: IEEE proceedings of international conference on affective computing & intelligent interaction, pp 349–355Google Scholar
  40. 40.
    Griffin HJ, Aung MSH, Romera-Paredes B, McLoughlin C, McKeown G, Curran W, Bianchi-Berthouze N (2015) Perception and automatic recognition of laughter from whole-body motion: continuous and categorical perspectives. IEEE Transact Affect Comput 6:165–178CrossRefGoogle Scholar
  41. 41.
    Mancini M, Varni G, Niewiadomski R, Volpe G, Camurri A (2014) How is your laugh today? In: CHI’14 extended abstracts on human factors in computing systems, pp 1855–1860Google Scholar
  42. 42.
    Niewiadomski R, Mancini M, Varni G, Volpe G, Camurri A (n.d.) Automated laughter detection from full-body movements. IEEE Transactions on Human-Machine Systems (in press)Google Scholar
  43. 43.
    Mancini M, Ach L, Bantegnie E, Baur T, Bianchi-Berthouze N, Datta D, Ding Y, Dupont S, Griffin HJ, Lingenfelser F, Niewiadomski R, Pelachaud C, Pietquin O, Piot B, Urbain J, Volpe G, Wagner J (2014) Laugh when you’re winning, innovative and creative developments in multimodal interaction systems. Springer, Berlin, pp 50–79CrossRefGoogle Scholar
  44. 44.
    Niewiadomski R, Mancini M, Ding Y, Pelachaud C, Volpe G (2014) Rhythmic body movements of laughter. In: Proceedings of the 16th international conference on multimodal interaction, pp 299–306Google Scholar
  45. 45.
    Griffin H, Varni G, Tome-Lourido G, Mancini M, Volpe G, Bianchi-Berthouze N (2015) Gesture mimicry in expression of laughter. ACII’15Google Scholar
  46. 46.
    Niedenthal PM (2007) Embodying emotion. Science 316(5827):1002–1005CrossRefPubMedGoogle Scholar
  47. 47.
    Riskind JH, Gotay CC (1982) Physical posture: could it have regulatory or feedback effects on motivation and emotion? Motiv Emot 6(3):273–298CrossRefGoogle Scholar
  48. 48.
    Brinol P, Petty R, Wagner B (2009) Body posture effect on self-evaluation: a self-validated approach. Eur J Soc Psychol 39:1053–1064CrossRefGoogle Scholar
  49. 49.
    Cacioppo JT, Priester JR, Berntson GG (1993) Rudimentary determinants of attitudes: II. Arm flexion and extension have differential effects on attitudes. J Pers Soc Psychol 65:5–17CrossRefPubMedGoogle Scholar
  50. 50.
    Casasanto D, Dijkstra K (2010) Motor action and emotional memory. Cognition 115(1):179–185CrossRefPubMedPubMedCentralGoogle Scholar
  51. 51.
    Wells GL, Petty RE (1980) The effects of overt head movements on persuasion: compatibility and incompatibility of responses. Basic Appl Soc Psychol 1:219–230CrossRefGoogle Scholar
  52. 52.
    Carney DR, Cuddy AJC, Yap AJ (2010) Power posing: brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychol Sci 21(10):1363–1368CrossRefPubMedGoogle Scholar
  53. 53.
    Botvinick M, Cohen J (1998) Rubber hands ‘feel’ touch that eyes see. Nature 391:756, Issue 39 CrossRefPubMedGoogle Scholar
  54. 54.
    Tsakiris M (2010) My body in the brain: a neurocognitive model of body ownership. Neuropsychologia 48(3):703–712CrossRefPubMedGoogle Scholar
  55. 55.
    Tajadura-Jimenez A, Tsakiris M, Marquardt T, Bianchi-Berthouze N (2015) Action sounds update the mental representation of arm dimension: contributions of kinaesthesia and agency. Front Psychol 6:689PubMedPubMedCentralGoogle Scholar
  56. 56.
    Lindley SE, Le Couteur J, Bianchi-Berthouze N (2008) Stirring up experience through movement in game play: effects on engagement and social behaviour. In: SIGCHI conference on human factors in computing systems, pp 511–514Google Scholar
  57. 57.
    Berthouze NK, Kim W, Darshak P (2007) Does body movement engage you more in digital game play? And why? Lect Notes Comput Sci LNCS 4738:102–113CrossRefGoogle Scholar
  58. 58.
    Pasch M, Bianchi-Berthouze N, van Dijk B, Nijholt A (2009) Movement-based sports video games: investigating motivation and gaming experience. Entertain Comput 1(2):49–61CrossRefGoogle Scholar
  59. 59.
    Nijhar J, Bianchi-Berthouze N, Boguslawski G (2012) Does movement recognition precision affect the player experience in exertion games? Int Conf Intell Technol Interact Entertain (INTETAIN) LNICST 78:73–82Google Scholar
  60. 60.
    Melzer A, Derks I, Heydekorn J, Steffgen G (2010) Click or strike: realistic versus standard game controls in violent video games and their effects on agression. In: Yang HS, Malaka R, Hoshino J, Han JH (eds) 9th International Conference, ICEC 2010. Springer, BerlinGoogle Scholar
  61. 61.
    Isbister K, Schwekendiek U, Frye J (2011a) Wriggle: an exploration of emotional and social effects of movement. In: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pp 1885–1890Google Scholar
  62. 62.
    Bianchi-Berthouze N (2013) Understanding the role of body movement in player engagement. Hum Comput Interact 28(1):42–75Google Scholar
  63. 63.
    Singh A, Piana S, Pollarolo D, Volpe G, Varni G, Tajadura- Jiménez A, Williams A CdeC, Camurri A, Bianchi-Berthouze N (n.d.) Go-with-the-flow: tracking, analysis and sonification of movement and breathing to build confidence in activity despite chronic pain. HCI (in press)Google Scholar
  64. 64.
    Singh A, Klapper A, Jia J, Fidalgo A, Tajadura-Jimenez A, Kanakam N, Bianchi-Berthouze N, Williams A (2014) Motivating people with chronic pain to do physical activity: opportunities for technology design. In: Proceedings of the 32nd ACM conference on human factors in computing systems, pp 803–2012Google Scholar
  65. 65.
    Bresin R, de Witt A, Papetti S, Civolani M, Fontana F (2010) Expressive sonification of footstep sounds. Proceedings of ISonGoogle Scholar
  66. 66.
    Tajadura Jimenez A, Basia M, Deroy O, Fairhurst M, Marquardt N, Bianchi-Berthouze N (2015) As light as your footsteps: altering walking sounds to change perceived body weight, emotional state and gait. In: Proceedings of the SIGCHI conference on human factors in computing systemGoogle Scholar
  67. 67.
    Hatfield E, Cacioppo JT, Rapson RL (1994) Emotional contagion. Cambridge, Cambridge University PressGoogle Scholar
  68. 68.
    Marsh KL, Richardson MJ, Schmidt RC (2009) Social connection through joint action and interpersonal coordination. Top Cogn Sci 1(2):320CrossRefPubMedGoogle Scholar
  69. 69.
    Valdesolo P, DeSteno D (2011) Synchrony and the social tuning of compassion. Emotion 11(2):262CrossRefPubMedGoogle Scholar
  70. 70.
    Hall ET (1968) Proximics. Curr Anthropol 9(2–3):83CrossRefGoogle Scholar
  71. 71.
    Knapp ML, Hall JA (2002) Nonverbal communication in human interaction, 3rd edn. Holt, Rinehart & Winston, New YorkGoogle Scholar
  72. 72.
    Huizinga J (1955) Homo Ludens: a study of the play-element in culture. Beacon Press, BostonGoogle Scholar
  73. 73.
    Salen K, Zimmerman E (2004) Rules of play: game design fundamentals. MIT Press, MassachusettsGoogle Scholar
  74. 74.
    Stenros J, Paavilainen J, Mäyrä F (2009) The many faces of sociability and social play in games. In: Proceedings of the 13th International MindTrek Conference: everyday life in the Ubiquitous Era, MindTrek’09. ACM, New York, pp 82–89Google Scholar
  75. 75.
    Simon B (2009) Wii are out of control: bodies, game screens and the production of gestural excess. Loading 3(4).
  76. 76.
    Mueller F, Isbister K (2014) Movement-based game guidelines. In: Proceedings of the SIGCHI conference on human factors in computing systemsGoogle Scholar
  77. 77.
    Márquez Segura E, Isbister K (2015) Enabling co-located physical social play: a framework for design and evaluation. In: Bernhaupt R (ed) Game user experience evaluation, Springer, Switzerland, pp 209–238.
  78. 78.
    Reeves S (2011) Designing interfaces in public settings: understanding the role of the spectator. Human-computer interaction. Springer, BerlinCrossRefGoogle Scholar
  79. 79.
    Isbister K, Höök K (2009) On being supple: in search of rigor without rigidity in meeting new design and evaluation challenges for HCI practitioners. In: Proceedings of the SIGCHI conference on human factors in computing systemsGoogle Scholar
  80. 80.
    Huggard A, De Mel A, Garner J, Toprak C, Chatham A, Mueller F (2013) Musical embrace: understanding a socially awkward digital play journey. DiGRA, Atlanta, GeorgiaGoogle Scholar
  81. 81.
    Garner J, Wood G, Pijnappel S, Murer M, Mueller F (2014) I-dentity: innominate movement representation as engaging game element. In: Proceedings of the SIGCHI conference on human factors in computing systemsGoogle Scholar
  82. 82.
    Isbister K (2012) How to stop being a buzzkill: designing Yamove!, a mobile tech mash-up to truly augment social play. MobileHCI’12Google Scholar
  83. 83.
    Robbins H, Isbister K (2014) Pixel motion: a surveillance camera enabled public digital game. In: Proceedings of foundations of digital gameGoogle Scholar
  84. 84.
    Moreno A, van Delden R, Poppe R, Reidsma D (2013) Socially aware interactive playgrounds. Pervasive Comput 12:40–47CrossRefGoogle Scholar
  85. 85.
    Huis in t’Veld EMJ, Van Boxtel GJM, de Gelder B (2014) The body action coding system I: muscle activations during perception and activation of emotion. Soc Neurosci 9(3):249–264CrossRefGoogle Scholar
  86. 86.
    Huis in t’Veld EMJ, Van Boxtel GJM, de Gelder B (2014) The body action coding system II: muscle activations during perception and activation of emotion. Behav Neurosci 8:330Google Scholar
  87. 87.
    Hertenstein MJ, Holmes R, Mccullough M, Keltner D (2009) The communication of emotion via touch. Emotion 9(4):566–573CrossRefPubMedGoogle Scholar
  88. 88.
    Atkinson D, Orzechowski P, Petreca B, Bianchi-Berthouze N, Watkins P, Baurley S, Padilla S, Chantler M (2013) Tactile perceptions of digital textiles: a design research approach. In: Proceedings of the SIGCHI conference on human factors in computing systemsGoogle Scholar
  89. 89.
    Sae-Bae N, Ahmed K, Isbister K, Memon N (2012) Biometric-rich gestures: a novel approach to authentication on multi-touch devices. In:Proceedings of the SIGCHI conference on human factors in computing systemsGoogle Scholar
  90. 90.
    Gao Y, Bianchi-Berthouze N, Meng H (2012) What does touch tell us about emotions in touchscreen-based gameplay? ACM Transact Comput Hum Interact 19(14):31Google Scholar
  91. 91.
    Siddiquie B, Amer M, Tamrakar A, Salter D, Lande B, Mehri D, Divakaran A (2015) Tower game dataset: a multimodal dataset for analyzing social interaction predicates. ACII’15Google Scholar
  92. 92.
    Lund HH, Klitbo T, Jessen C (2005) Playware technology for physically activating play. Artif Life Robot J 9:165–174CrossRefGoogle Scholar
  93. 93.
    Yannakakis GN, Hallam J (2008) Entertainment modeling through physiology in physical play. Int J Hum-Comput Stud 66(10):741–755CrossRefGoogle Scholar
  94. 94.
    Volkova E (2014) Expressions in narrative scenarios and across cultures. PhD thesis, Tubingen University.

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.UCLIC University College LondonLondonUK
  2. 2.Computational MediaUniversity of CaliforniaSanta CruzUSA

Personalised recommendations