From Sinewaves to Physiologically-Adaptive Soundscapes: The Evolving Relationship Between Sound and Emotion in Video Games

  • Tom A. Garner
Part of the Socio-Affective Computing book series (SAC, volume 4)


This chapter examines the dynamic and evolving relationship between sound and emotion within the context of video games. How sound in games has been utilised to both infer and evoke emotion is discussed, commencing with an historical review that traces back to video games’ humble beginnings. As we move towards the present day this chapter looks at how biofeedback technology, that can facilitate the control and procedural generation of game sound content by way of player-emotion, is transforming the lateral affective interplay between player and video game into something more circular.


Evoking emotion Pulse-code modulation-PCM Affective experience Psychophysiology Biofeedback 


  1. 1.
    Alves V, Roque L (2009) A proposal of soundscape design guidelines for user experience enrichment. Audio Mostly 2009. September 2nd-3rd, GlasgowGoogle Scholar
  2. 2.
    Amon K, Campbell A (2008) Can children with ADHD learn relaxation and breathing techniques through biofeedback video games? Aust J Educ Dev Psychol 8:72–84Google Scholar
  3. 3.
    AMD (2015) TrueAudio. Retrieved 08 July 2015
  4. 4.
  5. 5.
    Boris D (1998) Odyssey 2 Technical Specs Retrieved 25 June 2015
  6. 6.
    Bradley MM, Lang P (2000) Affective reactions to acoustic stimuli. Psychophysiology 37:204–215CrossRefPubMedGoogle Scholar
  7. 7.
    Breinbjerg M (2005) The aesthetic experience of sound – staging of auditory spaces in 3D computer games. Aesthetics of Play. Bergen, Norway, October 14th–15thGoogle Scholar
  8. 8.
    Buchanan K (2013) You’ll never guess how the dinosaur sounds in Jurassic Park were made. Retrieved 07 July 2015
  9. 9.
    Cacioppo JT, Tassinary LG (1990) Principles of psychophysiology: physical, social, and inferential elements. Cambridge University Press, New YorkGoogle Scholar
  10. 10.
    Cantor D (1971) A computer program that accepts common musical notation. Comput Hum 6(2):103–109CrossRefGoogle Scholar
  11. 11.
    Chanel G, Kierkels JJM, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Hum Comput Stud 67(8):607–662CrossRefGoogle Scholar
  12. 12.
    Chang K, Kim G, Kim T (2007, August) Video game console audio: evolution and future trends. In Computer graphics, imaging and visualisation, 2007. CGIV’07Google Scholar
  13. 13.
    Cho J, Yi E, Cho G (2001) Physiological responses evoked by fabric sounds and related mechanical and acoustical properties. Text Res J 71(12):1068–1073CrossRefGoogle Scholar
  14. 14.
    Cole T (2015) The tragedy of betrayal: how the design of Ico and Shadow of the Colossus elicits emotionGoogle Scholar
  15. 15.
    Collins K (2008) Game sound: an introduction to the history, theory, and practice of video game music and sound design. MIT PressGoogle Scholar
  16. 16.
    Collins K (2011) Making gamers cry: mirror neurons and embodied interaction with game sound. In Proceedings of the 6th audio mostly conference: a conference on interaction with sound, 39–46. ACMGoogle Scholar
  17. 17.
    Connor S (2010) Suspense-building music mimicks sounds of animals in distress. Independent Online. Retrieved 07 July 2015
  18. 18.
    Critchley HD, Mathias CJ, Dolan RJ (2002) Fear conditioning in humans: the influence of awareness and autonomic arousal on functional neuroanatomy. Neuron 33:653–663CrossRefPubMedGoogle Scholar
  19. 19.
    Dekker A, Champion E (2007) Please biofeed the zombies: enhancing the gameplay and display of a horror game using biofeedback. Proceedings of DiGRA, 550–558Google Scholar
  20. 20.
    Dixon MJ, MacLaren V, Jarick M, Fugelsang JA, Harrigan KA (2013) The frustrating effects of just missing the jackpot: slot machine near-misses trigger large skin conductance responses, but no post-reinforcement pauses. J Gambl Stud 29(4):661–674CrossRefPubMedGoogle Scholar
  21. 21.
    Drachen A, Nacke LE, Yannakakis G, Pedersen AL (2010) Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. 5th ACM SIGGRAPH Symposium on Video Games. 49–54. ACMGoogle Scholar
  22. 22.
    Duggan M (2008) Torque for teens. Cengage LearningGoogle Scholar
  23. 23.
    Ekman I (2008) Psychologically motivated techniques for emotional sound in computer games. Audio Mostly 2008 (Pitea, Sweden)Google Scholar
  24. 24.
    Fischer G, Grudin J, Lemke A et al (1992) Supporting indirect collaborative design with integrated knowledge-based design environments. Hum Comput Interact 7(3):281–314CrossRefGoogle Scholar
  25. 25.
    Gilroy SW, Porteous J, Charles F, Cavazza MO (2012) Exploring passive user interaction for adaptive narratives. Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces. Lisbon, Portugal, 14th–17th February 2012, ACM, New York, pp 119–128Google Scholar
  26. 26.
    Gonzalez-Sanchez J, Chavez-Echeagaray ME, Atkinson R, Burleson W (2011) ABE: an agent based software architecture for a multimodal emotion recognition framework. In: Proceedings of the ninth working IEEE/IFIP conference on software architecture. pp 187–193Google Scholar
  27. 27.
    Graetz JM (1981) The origin of spacewar. Creat Comput 7(8):56–67Google Scholar
  28. 28.
    Grimshaw M, Garner T (2015) Sonic virtuality: sound as emergent perception. Oxford University Press, New YorkCrossRefGoogle Scholar
  29. 29.
    Grimshaw M, Lindley CA, Nacke L (2008) Sound and immersion in the first-person shooter: mixed measurement of the player's sonic experience. Audio Mostly 2008, Piteå, SwedenGoogle Scholar
  30. 30.
    Hamalainen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounesmaa OV (1993) Magnetoencephalography: theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev Mod Physiol 65:413–497CrossRefGoogle Scholar
  31. 31.
    Hamilton MF, Blackstock DT (1998) Nonlinear acoustics, vol 427. Academic, San DiegoGoogle Scholar
  32. 32.
    Headlee K, Koziupa T, Siwiak D (2010) Sonic virtual reality game: How does your body sound? In: International conference on new interfaces for musical expression, Sydney, Australia, pp 15–18Google Scholar
  33. 33.
    Hermann T, Hunt A, Neuhoff (2011) The sonification handbook. Logos Verlag, BerlinGoogle Scholar
  34. 34.
    Huron DB (2006) Sweet anticipation: music and the psychology of expectation. MIT PressGoogle Scholar
  35. 35.
    Ismail F, Biedert R, Dengel A, Buscher G (2011) Emotional text tagging.
  36. 36.
    Jennett C, Cox AL, Cairns P et al (2008) Measuring and defining the experience of immersion in games. Int J Hum Comput Stud 66(9):641–661CrossRefGoogle Scholar
  37. 37.
    Jørgensen K (2006) On the functional aspects of computer game audio. Audio-Mostly 2006, Pitea, SwedenGoogle Scholar
  38. 38.
    Koelsch S, Kilches S, Steinbeis N, Schelinski S (2008) Effects of unexpected chords and of performer’s expression on brain responses and electrodermal activity. PLoS ONE 3(7)Google Scholar
  39. 39.
    Kuikkaniemi K, Laitinen T, Turpeinen M, Saari T, Kosunen I, Ravaja N (2010) The influence of implicit and explicit biofeedback in first-person shooter games. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, pp 859–868Google Scholar
  40. 40.
    Lang PJ, Bradley MM, Cuthbert BN, Patrick CJ (1993) Emotion and psychopathology: a startle probe analysis. Prog Exp Pers Psychopathol Res 16:163–199PubMedGoogle Scholar
  41. 41.
    Linzmayer O (1983) The voice of odyssey. Creat Comput Video Arcade Games 1(1):60Google Scholar
  42. 42.
    Lopes P, Liapis A, Yannakakis GN (2015) Sonancia: sonification of procedurally Generated Game Levels. Retrieved 09 July 2015
  43. 43.
    McAllister N (2011) Biometrics – the future of video games? Edge Online, Retrieved 11 Feb 2014
  44. 44.
    Mirza-Babaei P, Long S, Foley E, McAllister G (2011) Understanding the contribution of biometrics to games user research, DiGRA, pp 329–347Google Scholar
  45. 45.
    Moncrieff S, Venkatesh S, Dorai C (2001) Affect computing in film through sound energy dynamics International Multimedia Conference. Ninth ACM Int Conf Multimed 9:525–527CrossRefGoogle Scholar
  46. 46.
    Murugappan M, Ramachandran N, Sazali Y (2009) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3(4):390–396CrossRefGoogle Scholar
  47. 47.
    Nacke L, Ambinder M, Canossa A, Mandryk R, Stach T (2009) Game metrics and biometrics: the future of player experience research. Future Play, 2009Google Scholar
  48. 48.
    Nacke LE, Kalyn M, Lough C, Mandryk RL (2011) Biofeedback game design: using direct and indirect physiological control to enhance game interaction. In: SIGCHI conference on human factors in computing systems. pp 103–112Google Scholar
  49. 49.
    Oculus VR (2016) Accessed 29 Sep 2016
  50. 50.
    Parker JR, Heerema J (2008) Audio interaction in computer mediated games. Int J Comput Games Technol 2008:1CrossRefGoogle Scholar
  51. 51.
    Perron B (2004) Sign of a threat: the effects of warning systems in survival horror games. Cosign 2004 Proceedings. Art Academy, University of Split, pp 132–141Google Scholar
  52. 52.
    Plans E, Morelli D, Plans D (2015) AudioNode: prototypical affective modelling in experience-driven procedural music generation. Retrieved 09 July 2015
  53. 53.
    Poh MZ, Loddenkemper T, Swenson NC, Goyal S, Madsen JR, Picard RW (2010) Continuous monitoring of electrodermal activity during epileptic seizures using a wearable sensor. In: Conf Proc IEEE Eng Med Biol Soc. pp 4415–4418Google Scholar
  54. 54.
    Ravaja N (2004) Contributions of psychophysiology to media research: review and recommendations. Media Psychol 6:193–235CrossRefGoogle Scholar
  55. 55.
    Sato W, Fujimura T, Suzuki N (2008) Enhanced facial EMG activity in response to dynamic facial expressions. Int J Psychophysiol 70(1):70–74CrossRefPubMedGoogle Scholar
  56. 56.
    Shilling R, Zyda M, Wardynski EC (2002) Introducing emotion into military simulation and video game design America’s Army: operations and VIRTE. GAME-ONGoogle Scholar
  57. 57.
    SPC700 APU Manual (1997) Retrieved 03 July 2015
  58. 58.
    Steckler L (1975) TV Games Al ‘I-IomeGoogle Scholar
  59. 59.
    Stony Brook University (2011) Tennis for two: history, Retrieved 02 July 2015
  60. 60.
    Sweet M (2014) Writing interactive music for video games: a composer’s guide. Pearson EducationGoogle Scholar
  61. 61.
    Tan ES (2000) Emotion, art, and the humanities. In: Lewis M, Haviland-Jones JM (eds) Handbook of emotions, 2nd edn. Guilford Press, New York, pp 116–134Google Scholar
  62. 62.
    Väisänen J, Väisänen O, Malmivuo J, Hyttinen J (2008) New method for analysing sensitivity distributions of electroencephalography measurements. Med Biol Eng Comput 46:101–108CrossRefPubMedGoogle Scholar
  63. 63.
    Video Game Consile Library (2014) Retrieved 20 June 2015
  64. 64.
    Visch VT, Tan ES, Molenaar D (2010) The emotional and cognitive effect of immersion in film viewing. Cognit Emot 24(8):1439–1445CrossRefGoogle Scholar
  65. 65.
    Winter D (2013) Atari PONG: the first steps, Retrieved 01 July 2015
  66. 66.
    Woodward JD, Orlans NM, Higgins PT (2003) Biometrics: [identity assurance in the information age]. McGraw-Hill/Osborne, New YorkGoogle Scholar
  67. 67.
    Xu M, Chia L, Jin J (2005) Affective content analysis in comedy and horror videos by audio emotional event detection. In: IEEE International Conference on Multimedia and ExpoGoogle Scholar
  68. 68.
    Yokota T, Fujimori B (1962) Impedence change of the skin during the galvanic skin reflex. Jpn J Physiol 12:200–209CrossRefPubMedGoogle Scholar
  69. 69.
    Yoo MJ, Beak JW, Lee IK (2011) Creating musical expression using kinect. In: Proceedings of New Interfaces for Musical Expression. Oslo, NorwayGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.School of Creative and Cultural IndustriesUniversity of PortsmouthPortsmouthUK

Personalised recommendations